2024-01-29T15:40:35.462 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-29T15:40:35.462 INFO:teuthology.results:subset: '512/20000' 2024-01-29T15:40:35.462 INFO:teuthology.results:seed: '2470' 2024-02-05T04:20:24.615 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-05T04:20:24.645 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-05T06:30:36.199 INFO:teuthology.results:Tests finished! gathering results... 2024-02-05T06:30:52.165 INFO:scrape:Found 200 jobs 2024-02-05T06:30:54.308 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7fceba254db0] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7fcebacd49c4] 2024-02-05T06:30:54.308 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-05T06:30:54.319 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7fceba254db0] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7fcebacd49c4] 2024-02-05T06:30:54.319 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-05T06:30:54.328 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7fceba254db0] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7fcebacd49c4] 2024-02-05T06:30:54.328 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-05T06:30:54.338 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7fceba254db0] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7fcebacd49c4] 2024-02-05T06:30:54.338 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-05T06:30:54.349 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7fceba254db0] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7fcebacd49c4] 2024-02-05T06:30:54.349 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-05T06:30:54.359 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7fceba254db0] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7fcebacd49c4] 2024-02-05T06:30:54.359 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-05T06:30:54.369 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7fceba254db0] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7fcebacd49c4] 2024-02-05T06:30:54.370 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-05T06:30:54.380 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7fceba254db0] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7fcebacd49c4] 2024-02-05T06:30:54.381 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-05T06:30:54.391 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7fceba254db0] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7fcebacd49c4] 2024-02-05T06:30:54.391 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-05T06:30:55.229 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536080/teuthology.log 2024-02-05T06:30:55.229 INFO:scrape:Found 11 distinct failure reasons 2024-02-05T06:30:55.229 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi188 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' 2024-02-05T06:30:55.229 INFO:scrape:1 jobs: ['7536404'] 2024-02-05T06:30:55.230 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/scrub}'] 2024-02-05T06:30:55.230 INFO:scrape: 2024-02-05T06:30:55.230 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2024-02-05T06:30:55.230 INFO:scrape:1 jobs: ['7536288'] 2024-02-05T06:30:55.230 INFO:scrape:suites: ['debug/mgr', 'mon_election/classic', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-zlib}', 'tasks/e2e}'] 2024-02-05T06:30:55.230 INFO:scrape: 2024-02-05T06:30:55.230 INFO:scrape:Failure: Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) 2024-02-05T06:30:55.230 INFO:scrape:1 jobs: ['7536152'] 2024-02-05T06:30:55.230 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'task/test_orch_cli}'] 2024-02-05T06:30:55.230 INFO:scrape: 2024-02-05T06:30:55.231 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-02-05T06:30:55.231 INFO:scrape:2 jobs: ['7536133', '7536277'] 2024-02-05T06:30:55.231 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'validater/valgrind}'] 2024-02-05T06:30:55.231 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-02-05T06:30:55.231 INFO:scrape: 2024-02-05T06:30:55.231 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2024-02-05T06:30:55.231 INFO:scrape:1 jobs: ['7536320'] 2024-02-05T06:30:55.231 INFO:scrape:suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}'] 2024-02-05T06:30:55.231 INFO:scrape: 2024-02-05T06:30:55.231 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2024-02-05T06:30:55.231 INFO:scrape:1 jobs: ['7536390'] 2024-02-05T06:30:55.232 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] 2024-02-05T06:30:55.232 INFO:scrape: 2024-02-05T06:30:55.232 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*/0 --gtest_catch_exceptions=0\'' 2024-02-05T06:30:55.232 INFO:scrape:1 jobs: ['7536374'] 2024-02-05T06:30:55.232 INFO:scrape:suites: ['rados/objectstore/{backends/objectstore-memstore', 'supported-random-distro$/{centos_latest}}'] 2024-02-05T06:30:55.232 INFO:scrape: 2024-02-05T06:30:55.232 INFO:scrape:Dead: b'2024-01-30T03:53:17.225 DEBUG:teuthology.exit:Finished running handlers' 2024-02-05T06:30:55.233 INFO:scrape:1 jobs: ['7536124'] 2024-02-05T06:30:55.233 INFO:scrape:suites: ['mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{centos_latest}}'] 2024-02-05T06:30:55.233 INFO:scrape: 2024-02-05T06:30:55.233 INFO:scrape:Failure: Command failed on smithi132 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' 2024-02-05T06:30:55.233 INFO:scrape:1 jobs: ['7536126'] 2024-02-05T06:30:55.233 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-02-05T06:30:55.233 INFO:scrape: 2024-02-05T06:30:55.233 INFO:scrape:Failure: SSH connection to smithi002 was lost: 'sudo yum install -y kernel' 2024-02-05T06:30:55.234 INFO:scrape:1 jobs: ['7536081'] 2024-02-05T06:30:55.234 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'task/test_cephadm_repos}'] 2024-02-05T06:30:55.234 INFO:scrape: 2024-02-05T06:30:55.234 INFO:scrape:Dead: None 2024-02-05T06:30:55.234 INFO:scrape:1 jobs: ['7536080'] 2024-02-05T06:30:55.234 INFO:scrape: 2024-02-05T06:30:55.240 INFO:teuthology.results:Sending results to pdiazbou@redhat.com: Test Run: pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/ failed: 9 dead: 3 running: 0 waiting: 0 queued: 0 passed: 188 Fail ================================================================= [7536126] rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:08:26 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536126/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536126/ sentry: https://sentry.ceph.com/organizations/ceph/?query=977e8e4e0b9e4b73ab67d68989b30711 Command failed on smithi132 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7536133] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:26:25 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536133/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536133/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7536152] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:23:17 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536152/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536152/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d87a9b2a9274115b51272d19f2f56d2 Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) [7536277] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:40:43 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536277/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536277/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7536288] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/e2e} ----------------------------------------------------------------- time: 00:28:58 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536288/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536288/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ebd0a0d3c364331a28fae9e086f6a48 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7536320] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:08:42 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536320/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536320/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3cb8880a9ad34005a3b4910fa99894c6 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7536374] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:07:58 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536374/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536374/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e02b482f4cb4d7e8d225448bf7431d7 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*/0 --gtest_catch_exceptions=0\'' [7536390] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:15:39 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536390/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee0215834dd94fa7b5f8fbca19d18887 Command failed (workunit test cephtool/test.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7536404] rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} ----------------------------------------------------------------- time: 02:05:47 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536404/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536404/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9d98677c75140bc92cb0a3e021baf6d Command failed (workunit test scrub/osd-scrub-test.sh) on smithi188 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' Dead ================================================================= [7536080] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536080/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536080/ Error reimaging machines: Expected smithi002's OS to be centos 9 but found rhel 8.6 [7536081] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536081/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536081/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cfc8644f193543f9b7c9af15ef9a810d SSH connection to smithi002 was lost: 'sudo yum install -y kernel' [7536124] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536124/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536124/ hit max job timeout Pass ================================================================= [7536077] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:15:30 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536077/ [7536078] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:14:40 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536078/ [7536079] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:19:53 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536079/ [7536082] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:23:53 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536082/ [7536083] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:09:06 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536083/ [7536084] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:26:13 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536084/ [7536085] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:25:05 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536085/ [7536086] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_mon_osdmap_prune} time: 00:44:31 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536086/ [7536087] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:26:01 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536087/ [7536088] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:58 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536088/ [7536089] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:18:48 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536089/ [7536090] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:17:43 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536090/ [7536091] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:34:44 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536091/ [7536092] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} time: 01:12:30 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536092/ [7536093] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:31:46 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536093/ [7536094] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:10:21 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536094/ [7536095] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados tasks/rados_cls_all validater/lockdep} time: 00:19:38 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536095/ [7536096] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:20:17 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536096/ [7536097] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:16:32 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536097/ [7536098] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:25 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536098/ [7536099] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:34:59 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536099/ [7536100] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-overwrites} time: 00:21:35 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536100/ [7536101] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_latest}} time: 00:23:27 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536101/ [7536102] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:24:25 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536102/ [7536103] rados/singleton/{all/peer mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:10:54 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536103/ [7536104] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:16:23 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536104/ [7536105] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:15:32 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536105/ [7536106] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:42 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536106/ [7536107] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:48 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536107/ [7536108] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:28 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536108/ [7536109] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:15:08 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536109/ [7536110] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_host_drain} time: 00:23:48 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536110/ [7536111] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:17:04 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536111/ [7536112] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:16:01 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536112/ [7536113] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:19:38 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536113/ [7536114] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:12:53 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536114/ [7536115] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:15:10 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536115/ [7536116] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:24:38 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536116/ [7536117] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:20 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536117/ [7536118] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_latest} tasks/crash} time: 00:10:14 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536118/ [7536119] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:15:42 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536119/ [7536120] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:15:18 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536120/ [7536121] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:20:12 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536121/ [7536122] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:40 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536122/ [7536123] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} tasks/dashboard} time: 01:22:06 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536123/ [7536125] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon-stretch} time: 00:23:36 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536125/ [7536127] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_mon_workunits} time: 00:27:56 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536127/ [7536128] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:20:14 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536128/ [7536129] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:27:00 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536129/ [7536130] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:08 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536130/ [7536131] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:14:02 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536131/ [7536132] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:08:28 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536132/ [7536134] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:19:03 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536134/ [7536135] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/basic 3-final} time: 00:13:07 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536135/ [7536136] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:59 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536136/ [7536137] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache} time: 00:10:47 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536137/ [7536138] rados/singleton/{all/radostool mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:15:20 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536138/ [7536139] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:21:56 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536139/ [7536140] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:18:12 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536140/ [7536141] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/test_rbd_api} time: 00:24:09 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536141/ [7536142] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:08:17 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536142/ [7536143] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:23:25 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536143/ [7536144] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-mixed} time: 00:14:16 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536144/ [7536145] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:53 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536145/ [7536146] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:31 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536146/ [7536147] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:18:25 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536147/ [7536148] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/dedup-io-snaps} time: 00:15:30 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536148/ [7536149] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:33:14 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536149/ [7536150] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:29:15 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536150/ [7536151] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:14:46 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536151/ [7536153] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{centos_latest}} time: 02:11:02 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536153/ [7536154] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} time: 00:25:44 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536154/ [7536155] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:14:33 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536155/ [7536156] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:17 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536156/ [7536157] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:11:36 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536157/ [7536159] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:18:47 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536159/ [7536161] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:27:52 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536161/ [7536164] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} time: 01:07:20 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536164/ [7536166] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:38:01 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536166/ [7536168] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:07:48 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536168/ [7536171] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_orch_cli_mon} time: 00:30:07 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536171/ [7536173] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/snaps-few-objects} time: 00:26:21 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536173/ [7536175] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-radosbench} time: 00:36:14 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536175/ [7536178] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:39 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536178/ [7536180] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:16:01 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536180/ [7536182] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench} time: 00:56:57 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536182/ [7536184] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/scrub_test} time: 00:09:46 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536184/ [7536187] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all} time: 00:14:49 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536187/ [7536189] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} time: 00:24:04 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536189/ [7536191] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:12 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536191/ [7536193] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/redirect} time: 00:17:23 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536193/ [7536195] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:34:49 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536195/ [7536197] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:08:34 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536197/ [7536200] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:18:36 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536200/ [7536202] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:12:06 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536202/ [7536204] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:14:52 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536204/ [7536207] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:07:12 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536207/ [7536209] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:29:14 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536209/ [7536211] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:23 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536211/ [7536213] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:18:45 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536213/ [7536216] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_set_object} time: 00:15:56 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536216/ [7536218] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:22:03 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536218/ [7536220] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:13:49 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536220/ [7536222] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:33:45 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536222/ [7536225] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:09:47 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536225/ [7536227] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:16:44 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536227/ [7536229] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:16:30 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536229/ [7536232] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:18:49 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536232/ [7536234] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:59 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536234/ [7536236] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:21:10 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536236/ [7536238] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:09:15 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536238/ [7536241] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:17:22 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536241/ [7536243] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:16:25 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536243/ [7536245] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd-backfill} time: 02:25:26 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536245/ [7536247] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:22:12 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536247/ [7536250] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:27:19 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536250/ [7536252] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:30 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536252/ [7536254] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_adoption} time: 00:12:11 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536254/ [7536256] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:52 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536256/ [7536259] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 00:08:41 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536259/ [7536261] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 01:54:08 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536261/ [7536263] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/many workloads/pool-create-delete} time: 00:18:54 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536263/ [7536265] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects} time: 00:21:46 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536265/ [7536268] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-balanced} time: 00:19:31 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536268/ [7536270] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:04 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536270/ [7536272] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:16:35 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536272/ [7536274] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:25:29 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536274/ [7536279] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:11 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536279/ [7536281] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:25:57 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536281/ [7536283] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_ca_signed_key} time: 00:19:32 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536283/ [7536286] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:27:58 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536286/ [7536290] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:25:33 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536290/ [7536293] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:24:01 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536293/ [7536295] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:05 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536295/ [7536297] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 1-start 2-services/rgw 3-final} time: 00:15:09 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536297/ [7536299] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:12:19 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536299/ [7536302] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:00 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536302/ [7536304] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:22:38 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536304/ [7536306] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:24 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536306/ [7536309] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/rados_api_tests} time: 00:42:52 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536309/ [7536311] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:15:12 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536311/ [7536313] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:18 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536313/ [7536315] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:59 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536315/ [7536318] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:09:08 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536318/ [7536322] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:17:50 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536322/ [7536324] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:21:34 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536324/ [7536327] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} time: 02:30:19 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536327/ [7536329] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_cephadm} time: 00:21:02 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536329/ [7536331] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 01:04:09 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536331/ [7536333] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:10:21 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536333/ [7536336] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:23:45 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536336/ [7536338] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:12:32 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536338/ [7536340] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:08:09 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536340/ [7536343] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_5925} time: 00:08:38 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536343/ [7536345] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read} time: 00:19:45 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536345/ [7536347] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:45 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536347/ [7536349] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:23:01 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536349/ [7536352] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_cephadm_repos} time: 00:07:32 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536352/ [7536354] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:19:57 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536354/ [7536356] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep} time: 00:11:18 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536356/ [7536358] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/rados_cls_all} time: 00:15:40 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536358/ [7536361] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:20:27 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536361/ [7536363] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:20:09 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536363/ [7536365] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:08:02 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536365/ [7536368] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:14:24 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536368/ [7536370] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:14:56 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536370/ [7536372] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:18:15 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536372/ [7536377] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:24:38 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536377/ [7536379] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:23:09 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536379/ [7536381] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:18:48 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536381/ [7536383] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:19:16 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536383/ [7536386] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:08:57 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536386/ [7536388] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:24:59 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536388/ [7536392] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:23:14 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536392/ [7536395] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:16 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536395/ [7536397] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:17:43 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536397/ [7536399] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache} time: 00:11:52 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536399/ [7536401] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:18:33 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536401/ [7536406] rados/singleton/{all/deduptool mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:11:09 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536406/ [7536408] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:32:31 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536408/ [7536410] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/dedup-io-mixed} time: 00:12:29 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536410/ [7536413] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_host_drain} time: 00:20:16 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536413/ [7536415] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:14:28 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536415/ [7536417] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:10 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536417/ [7536419] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} time: 00:15:45 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536419/ [7536422] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/rados_python} time: 00:14:45 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536422/ [7536424] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:21:25 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536424/ [7536426] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:11:40 info: https://pulpito.ceph.com/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536426/