2023-10-07T15:51:33.644 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-07T15:51:33.644 INFO:teuthology.results:subset: '111/120000' 2023-10-07T15:51:33.644 INFO:teuthology.results:seed: '4019' 2023-10-08T02:27:20.881 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-08T02:27:20.903 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-08T14:27:27.102 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-10-08T14:27:51.351 INFO:scrape:Found 307 jobs 2023-10-08T14:27:52.720 WARNING:scrape:Misunderstood line: 2023-10-08T14:27:53.405 WARNING:scrape:Misunderstood line: 2023-10-08T14:27:53.869 WARNING:scrape:Misunderstood line: 2023-10-08T14:27:55.335 WARNING:scrape:Misunderstood line: 2023-10-08T14:27:57.448 WARNING:scrape:Misunderstood line: 2023-10-08T14:27:57.448 INFO:scrape:Found 7 distinct failure reasons 2023-10-08T14:27:57.449 INFO:scrape:Dead: b'2023-10-08T09:55:55.363 DEBUG:teuthology.exit:Finished running handlers' 2023-10-08T14:27:57.449 INFO:scrape:3 jobs: ['7416965', '7416720', '7417102'] 2023-10-08T14:27:57.449 INFO:scrape:suites intersection: ['rados'] 2023-10-08T14:27:57.449 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_20.04}}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] 2023-10-08T14:27:57.449 INFO:scrape: 2023-10-08T14:27:57.449 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9725b08fd801c82987c737f7e6d354878f036ee8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2023-10-08T14:27:57.450 INFO:scrape:2 jobs: ['7416862', '7417106'] 2023-10-08T14:27:57.450 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'tasks/e2e}'] 2023-10-08T14:27:57.450 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'tasks/e2e}'] 2023-10-08T14:27:57.450 INFO:scrape: 2023-10-08T14:27:57.450 INFO:scrape:Crash: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' 2023-10-08T14:27:57.450 INFO:scrape:ceph version 18.0.0-6609-g9725b08f (9725b08fd801c82987c737f7e6d354878f036ee8) reef (dev) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f2effe6eafb] 2: (TierFlushOp::_finish(TestOp::CallbackInfo*)+0x39f) [0x55d3fea0d99f] 3: (write_callback(void*, void*)+0x1e) [0x55d3fea2aa7e] 4: /lib/librados.so.2(+0xbd25c) [0x7f2f009b625c] 5: /lib/librados.so.2(+0xd29a5) [0x7f2f009cb9a5] 6: /lib/librados.so.2(+0xd313a) [0x7f2f009cc13a] 7: /lib/librados.so.2(+0xc18a1) [0x7f2f009ba8a1] 8: /lib/librados.so.2(+0xd796f) [0x7f2f009d096f] 9: /usr/lib/ceph/libceph-common.so.2(+0xad6164) [0x7f2f00614164] 10: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f2eff84d609] 2023-10-08T14:27:57.450 INFO:scrape:1 jobs: ['7416668'] 2023-10-08T14:27:57.450 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_20.04}', 'thrashers/careful', 'thrashosds-health', 'workloads/dedup-io-snaps}'] 2023-10-08T14:27:57.451 INFO:scrape: 2023-10-08T14:27:57.451 INFO:scrape:Valgrind: mon (Leak_StillReachable) 2023-10-08T14:27:57.451 INFO:scrape:4 jobs: ['7416986', '7416900', '7416784', '7416636'] 2023-10-08T14:27:57.451 INFO:scrape:suites intersection: [] 2023-10-08T14:27:57.451 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] 2023-10-08T14:27:57.451 INFO:scrape: 2023-10-08T14:27:57.451 INFO:scrape:Failure: Command failed on smithi144 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'mkdir $TESTDIR/kvtest && cd $TESTDIR/kvtest && ceph_test_keyvaluedb'" 2023-10-08T14:27:57.451 INFO:scrape:3 jobs: ['7416882', '7416875', '7417052'] 2023-10-08T14:27:57.452 INFO:scrape:suites intersection: [] 2023-10-08T14:27:57.452 INFO:scrape:suites union: ['2-inject-leak/osd', 'centos_latest}', 'mon_election/classic', 'rados', 'rados/objectstore/{backends/keyvaluedb', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'rados/valgrind-leaks/{1-start', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}'] 2023-10-08T14:27:57.452 INFO:scrape: 2023-10-08T14:27:57.452 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2023-10-08T14:27:57.452 INFO:scrape:2 jobs: ['7416724', '7416967'] 2023-10-08T14:27:57.452 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-10-08T14:27:57.452 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_20.04}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-10-08T14:27:57.452 INFO:scrape: 2023-10-08T14:27:57.452 INFO:scrape:Failure: Command failed on smithi089 with status 32: 'sync && sudo umount -f /var/lib/ceph/osd/ceph-7' 2023-10-08T14:27:57.453 INFO:scrape:1 jobs: ['7416901'] 2023-10-08T14:27:57.453 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'validater/valgrind}'] 2023-10-08T14:27:57.453 INFO:scrape: 2023-10-08T14:27:57.462 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/ failed: 13 dead: 2 running: 1 waiting: 0 queued: 0 passed: 291 Fail ================================================================= [7416636] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:49:47 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416636/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416636/ saw valgrind issues [7416668] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_20.04} thrashers/careful thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:11:02 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416668/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416668/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' [7416724] rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:13:45 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416724/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416724/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9ebb3fae974b460fa9fc0fe5f82d9d24 Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi087 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7416784] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:46:00 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416784/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416784/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f896d7c0b0d4b20a3210e79264fe903 Command failed (workunit test rados/test.sh) on smithi069 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9725b08fd801c82987c737f7e6d354878f036ee8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7416862] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} tasks/e2e} ----------------------------------------------------------------- time: 00:31:03 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416862/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416862/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d0cf400292844ce992bd3656e39a2a8 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9725b08fd801c82987c737f7e6d354878f036ee8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7416875] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:11:14 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416875/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416875/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa320b65a07747b0a0d75ef3442a460b Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7416882] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:16:31 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416882/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416882/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec5f01ac19844da9bf3a13d9afae9134 Command failed on smithi144 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'mkdir $TESTDIR/kvtest && cd $TESTDIR/kvtest && ceph_test_keyvaluedb'" [7416900] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:21:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416900/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416900/ saw valgrind issues [7416901] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:34:52 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416901/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416901/ Command failed on smithi089 with status 32: 'sync && sudo umount -f /var/lib/ceph/osd/ceph-7' [7416967] rados/upgrade/parallel/{0-random-distro$/{ubuntu_20.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:43:27 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416967/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416967/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9aae64bca58641aba5bd375960675e49 Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi089 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7416986] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:36:18 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416986/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416986/ saw valgrind issues [7417052] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} ----------------------------------------------------------------- time: 00:22:28 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417052/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417052/ Command failed on smithi148 with status 32: 'sync && sudo umount -f /var/lib/ceph/osd/ceph-1' [7417106] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-lz4} tasks/e2e} ----------------------------------------------------------------- time: 00:29:20 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417106/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417106/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1276ea21ef6b473d8b893ed086c3b121 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi135 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9725b08fd801c82987c737f7e6d354878f036ee8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' Dead ================================================================= [7416720] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416720/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416720/ hit max job timeout [7416965] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416965/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416965/ hit max job timeout Running ================================================================= [7417102] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417102/ Pass ================================================================= [7416565] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/classic task/test_ca_signed_key} time: 00:15:08 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416565/ [7416567] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:25:00 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416567/ [7416568] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:16:30 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416568/ [7416570] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:13:49 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416570/ [7416573] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/lockdep} time: 00:25:15 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416573/ [7416575] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/rgw 3-final} time: 00:13:28 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416575/ [7416576] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:35 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416576/ [7416578] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/cache-pool-snaps} time: 00:21:58 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416578/ [7416580] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} time: 02:41:13 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416580/ [7416582] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:12:20 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416582/ [7416584] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:19:27 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416584/ [7416586] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:16:34 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416586/ [7416588] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04} tasks/scrub_test} time: 00:12:21 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416588/ [7416591] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/prometheus} time: 00:17:31 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416591/ [7416593] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_osdmap_prune} time: 00:48:01 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416593/ [7416594] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/connectivity task/test_cephadm} time: 00:16:50 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416594/ [7416596] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced} time: 00:29:19 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416596/ [7416598] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:34 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416598/ [7416600] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:34:53 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416600/ [7416602] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:17:41 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416602/ [7416604] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} tasks/mon_recovery} time: 00:13:14 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416604/ [7416606] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:31:30 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416606/ [7416608] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:23:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416608/ [7416610] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:16:37 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416610/ [7416612] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:28:10 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416612/ [7416614] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:19:39 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416614/ [7416616] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{centos_latest}} time: 00:08:14 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416616/ [7416618] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/classic task/test_cephadm_repos} time: 00:13:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416618/ [7416620] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} time: 00:20:15 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416620/ [7416622] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_20.04} thrashers/fastread thrashosds-health workloads/ec-small-objects} time: 00:21:16 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416622/ [7416624] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:11 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416624/ [7416626] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:20:23 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416626/ [7416628] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache} time: 00:15:55 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416628/ [7416630] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:20:20 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416630/ [7416632] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} time: 00:13:12 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416632/ [7416634] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:25:23 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416634/ [7416638] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:40 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416638/ [7416640] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/connectivity task/test_extra_daemon_features} time: 00:21:32 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416640/ [7416642] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:55 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416642/ [7416644] rados/standalone/{supported-random-distro$/{rhel_8} workloads/scrub} time: 02:45:20 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416644/ [7416646] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:09:46 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416646/ [7416648] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:40:43 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416648/ [7416650] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/dedup-io-mixed} time: 00:19:07 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416650/ [7416652] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:25:19 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416652/ [7416654] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} time: 00:09:42 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416654/ [7416656] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:18:23 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416656/ [7416659] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_20.04} thrashers/many workloads/rados_mon_workunits} time: 00:31:11 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416659/ [7416662] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:11:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416662/ [7416665] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:16:39 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416665/ [7416671] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:15:07 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416671/ [7416674] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:31:19 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416674/ [7416677] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:31:28 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416677/ [7416680] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:23:51 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416680/ [7416683] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:33:08 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416683/ [7416686] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:14:35 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416686/ [7416691] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/rados_api_tests} time: 00:26:47 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416691/ [7416693] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:29:00 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416693/ [7416695] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:21:42 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416695/ [7416696] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:12:02 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416696/ [7416698] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:28:19 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416698/ [7416700] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:17 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416700/ [7416702] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:31:35 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416702/ [7416704] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:49 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416704/ [7416706] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04}} time: 00:30:05 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416706/ [7416708] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/connectivity task/test_orch_cli} time: 00:16:05 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416708/ [7416710] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04} thrashers/mapgap thrashosds-health workloads/rados_api_tests} time: 00:27:45 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416710/ [7416712] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/dashboard} time: 01:25:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416712/ [7416714] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_8} tasks/crash} time: 00:14:36 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416714/ [7416716] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{rhel_8}} time: 00:43:04 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416716/ [7416718] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_20.04}} time: 00:13:22 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416718/ [7416722] rados/standalone/{supported-random-distro$/{centos_latest} workloads/c2c} time: 00:14:40 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416722/ [7416726] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:17:54 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416726/ [7416728] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:12:56 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416728/ [7416730] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:38 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416730/ [7416732] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:08:41 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416732/ [7416734] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:16:09 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416734/ [7416736] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:33:55 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416736/ [7416738] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:16:34 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416738/ [7416740] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:14:31 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416740/ [7416742] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/basic 3-final} time: 00:16:57 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416742/ [7416744] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/one workloads/snaps-few-objects} time: 00:40:47 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416744/ [7416746] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:14:26 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416746/ [7416748] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/rados_cls_all} time: 00:21:53 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416748/ [7416750] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:15:18 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416750/ [7416752] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:14:51 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416752/ [7416755] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/classic task/test_orch_cli_mon} time: 00:25:14 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416755/ [7416757] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} thrashers/none thrashosds-health workloads/radosbench} time: 00:25:16 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416757/ [7416759] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:26:20 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416759/ [7416761] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:11:05 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416761/ [7416763] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:13:20 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416763/ [7416765] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:01 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416765/ [7416767] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:19:17 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416767/ [7416768] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:15:58 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416768/ [7416769] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:32:47 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416769/ [7416770] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:13:02 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416770/ [7416771] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/connectivity task/test_rgw_multisite} time: 00:22:18 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416771/ [7416772] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:14 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416772/ [7416773] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_20.04} tasks/failover} time: 00:14:26 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416773/ [7416774] rados/standalone/{supported-random-distro$/{centos_8} workloads/crush} time: 00:25:02 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416774/ [7416775] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:14:06 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416775/ [7416776] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:51 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416776/ [7416778] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:11:41 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416778/ [7416780] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:16:45 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416780/ [7416782] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:32:07 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416782/ [7416786] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_20.04} tasks/rados_python} time: 00:16:05 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416786/ [7416788] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:11:10 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416788/ [7416790] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:17 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416790/ [7416792] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/classic task/test_set_mon_crush_locations} time: 00:23:24 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416792/ [7416794] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} time: 00:09:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416794/ [7416796] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:14:53 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416796/ [7416798] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:21:11 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416798/ [7416800] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:10:19 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416800/ [7416802] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:15:36 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416802/ [7416804] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:20:40 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416804/ [7416806] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:25:21 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416806/ [7416808] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:30:09 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416808/ [7416810] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:45 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416810/ [7416812] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/connectivity task/test_adoption} time: 00:15:12 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416812/ [7416814] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04}} time: 00:10:22 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416814/ [7416816] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:17:23 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416816/ [7416818] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:14:42 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416818/ [7416820] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:19:47 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416820/ [7416822] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:14:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416822/ [7416824] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/rados_stress_watch} time: 00:15:22 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416824/ [7416826] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:25 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416826/ [7416828] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:30:33 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416828/ [7416830] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04}} time: 00:17:17 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416830/ [7416832] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:20:51 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416832/ [7416834] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:20:42 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416834/ [7416836] rados/standalone/{supported-random-distro$/{rhel_8} workloads/erasure-code} time: 01:31:45 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416836/ [7416838] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:04 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416838/ [7416840] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/classic task/test_ca_signed_key} time: 00:16:19 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416840/ [7416842] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:34:25 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416842/ [7416844] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_20.04}} time: 00:19:09 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416844/ [7416846] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:13:57 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416846/ [7416848] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 00:17:24 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416848/ [7416850] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all} time: 00:17:23 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416850/ [7416852] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:19:43 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416852/ [7416854] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416854/ [7416856] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:15:35 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416856/ [7416858] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:18:09 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416858/ [7416860] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:17:08 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416860/ [7416864] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:14:55 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416864/ [7416865] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:19:41 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416865/ [7416866] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:22:45 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416866/ [7416867] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} time: 00:10:54 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416867/ [7416868] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/connectivity task/test_cephadm} time: 00:17:10 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416868/ [7416869] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/rados_striper} time: 00:13:00 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416869/ [7416870] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:15:38 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416870/ [7416871] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:16:15 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416871/ [7416873] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:33:06 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416873/ [7416877] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{rhel_8} tasks/module_selftest} time: 00:35:08 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416877/ [7416878] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 1-start 2-services/rgw 3-final} time: 00:17:11 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416878/ [7416880] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:29:41 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416880/ [7416884] rados/standalone/{supported-random-distro$/{ubuntu_20.04} workloads/mgr} time: 00:16:32 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416884/ [7416886] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:16:58 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416886/ [7416888] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/classic task/test_cephadm_repos} time: 00:10:18 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416888/ [7416890] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:22:04 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416890/ [7416892] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench} time: 01:59:55 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416892/ [7416894] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:12:53 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416894/ [7416895] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:25 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416895/ [7416896] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:26:16 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416896/ [7416897] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:18:58 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416897/ [7416898] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:18:30 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416898/ [7416899] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} time: 00:19:08 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416899/ [7416903] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:36:08 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416903/ [7416905] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:45 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416905/ [7416907] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:24:39 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416907/ [7416909] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:29:11 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416909/ [7416911] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:33:41 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416911/ [7416913] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:09:00 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416913/ [7416915] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/connectivity task/test_extra_daemon_features} time: 00:21:02 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416915/ [7416917] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:18:57 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416917/ [7416919] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_20.04} thrashers/force-sync-many workloads/rados_api_tests} time: 00:28:35 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416919/ [7416921] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:17:36 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416921/ [7416923] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:12:24 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416923/ [7416925] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:23:38 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416925/ [7416926] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:18:20 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416926/ [7416927] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:20:03 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416927/ [7416928] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04} tasks/mon_clock_with_skews} time: 00:08:31 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416928/ [7416929] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_20.04} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:55 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416929/ [7416930] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:19:04 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416930/ [7416931] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:25:25 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416931/ [7416932] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:17:00 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416932/ [7416933] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} time: 01:09:17 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416933/ [7416934] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:14:32 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416934/ [7416935] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04}} time: 00:32:01 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416935/ [7416936] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} time: 00:09:32 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416936/ [7416937] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} time: 00:20:27 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416937/ [7416938] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:25 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416938/ [7416939] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:43 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416939/ [7416940] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mix} time: 00:22:50 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416940/ [7416941] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_latest}} time: 00:23:44 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416941/ [7416942] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:16:40 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416942/ [7416943] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:18:41 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416943/ [7416944] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/connectivity task/test_orch_cli} time: 00:19:05 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416944/ [7416945] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:27:47 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416945/ [7416946] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:25:25 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416946/ [7416947] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:16:51 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416947/ [7416948] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:15:49 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416948/ [7416949] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:32 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416949/ [7416950] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:22:29 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416950/ [7416951] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:12:19 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416951/ [7416952] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:34:50 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416952/ [7416953] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/progress} time: 00:24:22 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416953/ [7416954] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:15:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416954/ [7416955] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:15:05 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416955/ [7416956] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:25:17 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416956/ [7416957] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:51:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416957/ [7416958] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:15:25 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416958/ [7416959] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:23:38 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416959/ [7416960] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/classic task/test_orch_cli_mon} time: 00:26:34 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416960/ [7416961] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:21:57 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416961/ [7416962] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:25:50 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416962/ [7416963] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:29:31 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416963/ [7416964] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} tasks/dashboard} time: 01:30:27 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416964/ [7416966] rados/standalone/{supported-random-distro$/{ubuntu_20.04} workloads/mon-stretch} time: 00:26:16 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416966/ [7416968] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:12:32 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416968/ [7416969] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/basic 3-final} time: 00:12:53 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416969/ [7416970] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:31:16 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416970/ [7416971] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:12:52 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416971/ [7416972] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:13:48 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416972/ [7416973] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} time: 00:21:05 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416973/ [7416974] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:39 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416974/ [7416975] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/connectivity task/test_rgw_multisite} time: 00:17:30 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416975/ [7416976] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:09:35 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416976/ [7416977] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:35:53 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416977/ [7416978] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:21:00 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416978/ [7416979] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:15:27 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416979/ [7416981] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:16:58 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416981/ [7416983] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:11:40 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416983/ [7416987] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:13:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416987/ [7416989] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 02:06:49 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416989/ [7416992] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:19:42 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416992/ [7416994] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} time: 02:21:38 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416994/ [7416996] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:21:08 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416996/ [7416997] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:15:09 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416997/ [7416999] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:18 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7416999/ [7417002] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:15:16 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417002/ [7417004] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:36:54 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417004/ [7417006] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/classic task/test_set_mon_crush_locations} time: 00:24:37 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417006/ [7417008] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits} time: 00:29:32 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417008/ [7417010] rados/standalone/{supported-random-distro$/{rhel_8} workloads/mon} time: 01:12:25 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417010/ [7417012] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache} time: 00:14:52 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417012/ [7417014] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:16:28 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417014/ [7417016] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:26 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417016/ [7417018] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/connectivity task/test_adoption} time: 00:17:48 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417018/ [7417020] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} time: 00:08:40 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417020/ [7417022] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:22:40 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417022/ [7417024] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:23:56 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417024/ [7417026] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all} time: 00:14:10 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417026/ [7417028] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/dedup-io-mixed} time: 00:13:41 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417028/ [7417030] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:41 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417030/ [7417032] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:24:25 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417032/ [7417034] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:34 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417034/ [7417036] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:27:15 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417036/ [7417038] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:18:56 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417038/ [7417040] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/repair_test} time: 00:21:46 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417040/ [7417042] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:14:57 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417042/ [7417044] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{rhel_8} tasks/workunits} time: 00:19:16 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417044/ [7417046] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/dedup-io-snaps} time: 00:19:49 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417046/ [7417048] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:15 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417048/ [7417050] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:44 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417050/ [7417054] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:13:33 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417054/ [7417056] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/classic task/test_ca_signed_key} time: 00:21:34 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417056/ [7417058] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04}} time: 00:09:37 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417058/ [7417060] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:19:54 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417060/ [7417062] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:24:57 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417062/ [7417064] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} time: 00:30:36 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417064/ [7417066] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:37 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417066/ [7417068] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} time: 00:09:52 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417068/ [7417070] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd-backfill} time: 02:30:05 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417070/ [7417072] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:34:41 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417072/ [7417074] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04} thrashers/sync-many workloads/pool-create-delete} time: 00:15:43 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417074/ [7417076] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:20:08 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417076/ [7417078] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{centos_latest}} time: 00:07:26 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417078/ [7417080] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:10:28 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417080/ [7417082] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:51 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417082/ [7417084] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:30:23 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417084/ [7417086] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04}} time: 00:20:13 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417086/ [7417088] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} time: 00:10:31 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417088/ [7417090] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/connectivity task/test_cephadm} time: 00:18:53 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417090/ [7417092] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:47 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417092/ [7417094] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/scrub_test} time: 00:16:25 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417094/ [7417096] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/classic task/test_cephadm_repos} time: 00:09:05 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417096/ [7417098] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:10:30 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417098/ [7417100] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:27:59 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417100/ [7417104] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} time: 00:24:27 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417104/ [7417108] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:13:35 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417108/ [7417110] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:31 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417110/ [7417112] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:28:20 info: http://pulpito.ceph.com/yuriw-2023-10-07_13:59:51-rados-main-distro-default-smithi/7417112/