2023-11-29T15:18:34.235 INFO:root:teuthology version: 0.0.1.dev212+g735a43c
2023-11-29T15:18:34.236 INFO:teuthology.results:subset: '512/20000'
2023-11-29T15:18:34.236 INFO:teuthology.results:seed: '14'
2023-11-29T15:57:59.435 INFO:root:teuthology version: 0.0.1.dev212+g735a43c
2023-11-29T15:57:59.438 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-30T03:58:58.922 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-11-30T03:59:06.221 INFO:scrape:Found 100 jobs
2023-11-30T03:59:06.273 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470719/teuthology.log
2023-11-30T03:59:10.104 INFO:scrape:Found 11 distinct failure reasons
2023-11-30T03:59:10.104 INFO:scrape:Dead: b'2023-11-30T03:31:03.389 DEBUG:teuthology.exit:Finished running handlers'
2023-11-30T03:59:10.105 INFO:scrape:11 jobs: ['7470708', '7470764', '7470741', '7470736', '7470716', '7470700', '7470725', '7470724', '7470744', '7470774', '7470706']
2023-11-30T03:59:10.105 INFO:scrape:suites intersection: ['openstack}', 'rados', 'thrashosds-health']
2023-11-30T03:59:10.105 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-small}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/redirect_promote_tests}', 'workloads/redirect}', 'workloads/set-chunks-read}', 'workloads/small-objects-localized}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}']
2023-11-30T03:59:10.105 INFO:scrape:
2023-11-30T03:59:10.105 INFO:scrape:Dead: b'2023-11-30T03:59:06.252 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops'
2023-11-30T03:59:10.105 INFO:scrape:3 jobs: ['7470799', '7470788', '7470795']
2023-11-30T03:59:10.105 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health']
2023-11-30T03:59:10.106 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashers/mapgap', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-snaps}', 'workloads/dedup-io-mixed}', 'workloads/dedup-io-snaps}']
2023-11-30T03:59:10.106 INFO:scrape:
2023-11-30T03:59:10.106 INFO:scrape:Dead: None
2023-11-30T03:59:10.106 INFO:scrape:4 jobs: ['7470719', '7470751', '7470767', '7470775']
2023-11-30T03:59:10.106 INFO:scrape:
2023-11-30T03:59:10.106 INFO:scrape:Failure: Found coredumps on ubuntu@smithi017.front.sepia.ceph.com
2023-11-30T03:59:10.106 INFO:scrape:1 jobs: ['7470722']
2023-11-30T03:59:10.106 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/libcephsqlite}']
2023-11-30T03:59:10.107 INFO:scrape:
2023-11-30T03:59:10.107 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi029 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=202cbbe4ffe6da43d7daba52b5ab4d579376b7f3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh'
2023-11-30T03:59:10.107 INFO:scrape:1 jobs: ['7470742']
2023-11-30T03:59:10.107 INFO:scrape:suites: ['debug/mgr', 'mon_election/classic', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-zstd}', 'tasks/e2e}']
2023-11-30T03:59:10.107 INFO:scrape:
2023-11-30T03:59:10.107 INFO:scrape:Failure: Command failed on smithi107 with status 1: 'sudo yum -y install ceph'
2023-11-30T03:59:10.107 INFO:scrape:2 jobs: ['7470760', '7470740']
2023-11-30T03:59:10.107 INFO:scrape:suites intersection: []
2023-11-30T03:59:10.107 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_ca_signed_key}', 'task/test_cephadm}']
2023-11-30T03:59:10.107 INFO:scrape:
2023-11-30T03:59:10.108 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats'
2023-11-30T03:59:10.108 INFO:scrape:1 jobs: ['7470756']
2023-11-30T03:59:10.108 INFO:scrape:suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}']
2023-11-30T03:59:10.108 INFO:scrape:
2023-11-30T03:59:10.108 INFO:scrape:Failure: Command failed on smithi022 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:202cbbe4ffe6da43d7daba52b5ab4d579376b7f3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f3d3a0c-8ece-11ee-95a2-87774f69a715 -- ceph rgw realm bootstrap -i -'
2023-11-30T03:59:10.108 INFO:scrape:1 jobs: ['7470703']
2023-11-30T03:59:10.108 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_rgw_multisite}']
2023-11-30T03:59:10.108 INFO:scrape:
2023-11-30T03:59:10.108 INFO:scrape:Failure: failed to install new kernel version within timeout
2023-11-30T03:59:10.108 INFO:scrape:1 jobs: ['7470752']
2023-11-30T03:59:10.109 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/write_fadvise_dontneed}']
2023-11-30T03:59:10.109 INFO:scrape:
2023-11-30T03:59:10.109 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json
2023-11-30T03:59:10.109 INFO:scrape:1 jobs: ['7470711']
2023-11-30T03:59:10.109 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/redirect_set_object}']
2023-11-30T03:59:10.109 INFO:scrape:
2023-11-30T03:59:10.109 INFO:scrape:Failure: valgrind error: Leak_StillReachable
operator new[](unsigned long)
allocate
allocate
2023-11-30T03:59:10.109 INFO:scrape:1 jobs: ['7470737']
2023-11-30T03:59:10.109 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/valgrind}']
2023-11-30T03:59:10.109 INFO:scrape:
2023-11-30T03:59:10.113 INFO:teuthology.results:Sending results to akupczyk@redhat.com: Test Run: akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/
failed:  9
dead:    15
running: 3
waiting: 0
queued:  0
passed:  73


Fail
=================================================================
[7470703]  rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_rgw_multisite}
-----------------------------------------------------------------
time:   00:24:53
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470703/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470703/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9ac1625546234bbba4da0a8d35ed1e70

    Command failed on smithi022 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:202cbbe4ffe6da43d7daba52b5ab4d579376b7f3 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    7f3d3a0c-8ece-11ee-95a2-87774f69a715 -- ceph rgw realm bootstrap -i -'

[7470711]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/redirect_set_object}
-----------------------------------------------------------------
time:   00:22:27
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470711/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470711/

    Command failed on smithi082 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph osd dump --format=json'

[7470722]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite}
-----------------------------------------------------------------
time:   00:19:42
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470722/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470722/

    Found coredumps on ubuntu@smithi017.front.sepia.ceph.com

[7470737]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind}
-----------------------------------------------------------------
time:   01:48:07
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470737/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470737/

    valgrind error: Leak_StillReachable operator new[](unsigned long) allocate
    allocate

[7470740]  rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_ca_signed_key}
-----------------------------------------------------------------
time:   00:16:11
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470740/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470740/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9bc9d90e635e41bdb67237eb1326d92a

    Command failed on smithi170 with status 1: 'sudo yum -y install ceph'

[7470742]  rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zstd} tasks/e2e}
-----------------------------------------------------------------
time:   00:39:34
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470742/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470742/
sentry: https://sentry.ceph.com/organizations/ceph/?query=69de5a1e2a344164a7b5fe27b13788e9

    Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi029
    with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd
    -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=202cbbe4ffe6da43d7daba52b5ab4d579376b7f3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh'

[7470752]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/write_fadvise_dontneed}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470752/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470752/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bd96c653d8ec4a8a94d3c913dad5237b

    failed to install new kernel version within timeout

[7470756]  rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:18:36
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470756/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470756/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8894a973e0614f7dbf8de18c4c612624

    Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c
    ceph_test_lazy_omap_stats'

[7470760]  rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_cephadm}
-----------------------------------------------------------------
time:   00:15:14
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470760/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470760/
sentry: https://sentry.ceph.com/organizations/ceph/?query=95500f6617334d15a932e57d6714dba0

    Command failed on smithi107 with status 1: 'sudo yum -y install ceph'



Dead
=================================================================
[7470700]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/redirect}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470700/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470700/

    hit max job timeout

[7470706]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470706/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470706/

    hit max job timeout

[7470708]  rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470708/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470708/

    hit max job timeout

[7470716]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470716/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470716/

    hit max job timeout

[7470719]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/small-objects-balanced}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470719/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470719/

    Error reimaging machines: reached maximum tries (101) after waiting for 600
    seconds

[7470724]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/small-objects-localized}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470724/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470724/

    hit max job timeout

[7470725]  rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470725/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470725/

    hit max job timeout

[7470736]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470736/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470736/

    hit max job timeout

[7470741]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470741/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470741/

    hit max job timeout

[7470744]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470744/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470744/

    hit max job timeout

[7470751]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470751/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470751/

    Error reimaging machines: list index out of range

[7470764]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-agent-small}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470764/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470764/

    hit max job timeout

[7470767]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470767/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470767/

    Error reimaging machines: list index out of range

[7470774]  rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470774/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470774/

    hit max job timeout

[7470775]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470775/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470775/

    Error reimaging machines: reached maximum tries (101) after waiting for 600
    seconds



Running
=================================================================
[7470788] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps}
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470788/

[7470795] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/dedup-io-mixed}
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470795/

[7470799] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps}
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470799/



Pass
=================================================================
[7470701] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}
time:   00:36:32
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470701/

[7470702] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}
time:   00:14:34
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470702/

[7470704] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write}
time:   00:14:19
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470704/

[7470705] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/rbd_cls}
time:   00:27:21
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470705/

[7470707] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews}
time:   00:12:30
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470707/

[7470709] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:19:48
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470709/

[7470710] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:22:14
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470710/

[7470712] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:32:53
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470712/

[7470713] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/insights}
time:   00:23:44
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470713/

[7470714] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}}
time:   00:18:22
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470714/

[7470715] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest}
time:   00:18:55
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470715/

[7470717] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_set_mon_crush_locations}
time:   00:26:41
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470717/

[7470718] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_latest}}
time:   00:10:05
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470718/

[7470720] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}}
time:   00:17:10
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470720/

[7470721] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:25:00
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470721/

[7470723] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd-backfill}
time:   02:27:19
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470723/

[7470726] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:14:28
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470726/

[7470727] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_adoption}
time:   00:21:06
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470727/

[7470728] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write}
time:   00:27:54
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470728/

[7470729] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{centos_latest}}
time:   00:09:27
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470729/

[7470730] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}}
time:   02:33:01
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470730/

[7470731] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/pool-create-delete}
time:   00:25:22
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470731/

[7470732] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects}
time:   00:34:01
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470732/

[7470733] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-balanced}
time:   00:23:09
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470733/

[7470734] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:23:25
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470734/

[7470735] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:22:42
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470735/

[7470738] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:21:25
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470738/

[7470739] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}}
time:   00:37:12
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470739/

[7470743] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_latest} tasks/module_selftest}
time:   00:31:54
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470743/

[7470745] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{centos_latest}}
time:   00:17:24
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470745/

[7470746] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final}
time:   00:20:31
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470746/

[7470747] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/mon_recovery}
time:   00:20:22
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470747/

[7470748] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:24:11
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470748/

[7470749] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}}
time:   00:28:32
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470749/

[7470750] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:37:39
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470750/

[7470753] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_fio}
time:   00:21:32
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470753/

[7470754] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:21:10
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470754/

[7470755] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/snaps-few-objects}
time:   00:44:36
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470755/

[7470757] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests}
time:   00:28:24
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470757/

[7470758] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}
time:   00:27:14
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470758/

[7470759] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd}
time:   02:43:44
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470759/

[7470761] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-agent-big}
time:   00:47:06
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470761/

[7470762] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}}
time:   00:14:56
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470762/

[7470763] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:31:13
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470763/

[7470765] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}
time:   00:20:43
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470765/

[7470766] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925}
time:   00:20:35
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470766/

[7470768] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:13:53
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470768/

[7470769] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy}
time:   00:25:02
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470769/

[7470770] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_cephadm_repos}
time:   00:08:10
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470770/

[7470771] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:20:56
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470771/

[7470772] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep}
time:   00:12:31
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470772/

[7470773] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all}
time:   00:23:42
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470773/

[7470776] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}
time:   00:11:03
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470776/

[7470777] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_latest} tasks/per_module_finisher_stats}
time:   00:10:28
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470777/

[7470778] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench}
time:   00:13:30
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470778/

[7470779] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:19:04
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470779/

[7470780] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{ubuntu_latest}}
time:   00:10:34
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470780/

[7470781] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-snaps-balanced}
time:   00:20:41
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470781/

[7470782] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:22:25
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470782/

[7470783] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_extra_daemon_features}
time:   00:18:22
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470783/

[7470784] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}}
time:   00:19:11
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470784/

[7470785] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews}
time:   00:09:24
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470785/

[7470786] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
time:   00:28:38
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470786/

[7470787] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}
time:   00:35:11
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470787/

[7470789] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:28:01
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470789/

[7470790] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:15:26
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470790/

[7470791] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache}
time:   00:14:36
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470791/

[7470792] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:21:11
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470792/

[7470793] rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub}
time:   02:34:25
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470793/

[7470794] rados/singleton/{all/deduptool mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}}
time:   00:12:11
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470794/

[7470796] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_host_drain}
time:   00:19:20
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470796/

[7470797] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read}
time:   00:15:29
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470797/

[7470798] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:12:04
info:   https://pulpito.ceph.com/akupczyk-2023-11-29_15:17:59-rados-aclamk-test-sb-to-b-vselector-2-distro-default-smithi/7470798/