2024-01-13T03:25:31.317 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-13T03:25:31.317 INFO:teuthology.results:subset: '111/120000'
2024-01-13T03:25:31.317 INFO:teuthology.results:seed: '726'
2024-01-13T05:59:45.727 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-13T05:59:45.762 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-01-13T07:02:51.208 INFO:teuthology.results:Tests finished! gathering results...
2024-01-13T07:03:02.407 INFO:scrape:Found 161 jobs
2024-01-13T07:03:06.565 INFO:scrape:Found 5 distinct failure reasons
2024-01-13T07:03:06.566 INFO:scrape:Failure: Command failed on smithi086 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c2c6010-b1d2-11ee-95ac-87774f69a715 -e sha1=1ec1182e55b4a38b17dd0d004d9fa6814d0269d4 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''
2024-01-13T07:03:06.566 INFO:scrape:4 jobs: ['7515250', '7515385', '7515091', '7515343']
2024-01-13T07:03:06.566 INFO:scrape:suites intersection: []
2024-01-13T07:03:06.566 INFO:scrape:suites union: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04']
2024-01-13T07:03:06.566 INFO:scrape:
2024-01-13T07:03:06.566 INFO:scrape:Failure: Command failed on smithi012 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"
2024-01-13T07:03:06.567 INFO:scrape:4 jobs: ['7515325', '7515244', '7515083', '7515164']
2024-01-13T07:03:06.567 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_20.04']
2024-01-13T07:03:06.567 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}']
2024-01-13T07:03:06.567 INFO:scrape:
2024-01-13T07:03:06.567 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi061 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ec1182e55b4a38b17dd0d004d9fa6814d0269d4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
2024-01-13T07:03:06.567 INFO:scrape:1 jobs: ['7515363']
2024-01-13T07:03:06.567 INFO:scrape:suites: ['1-start', '2-thrash', '3-tasks/rados_api_tests', 'fixed-2', 'msgr/async-v2only', 'orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun', 'root}']
2024-01-13T07:03:06.567 INFO:scrape:
2024-01-13T07:03:06.567 INFO:scrape:Failure: Command failed on smithi149 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1ec1182e55b4a38b17dd0d004d9fa6814d0269d4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a8d20ee-b1c8-11ee-95ac-87774f69a715 -- ceph mon dump -f json'
2024-01-13T07:03:06.567 INFO:scrape:1 jobs: ['7515122']
2024-01-13T07:03:06.568 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-user', '3-final}', 'orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8']
2024-01-13T07:03:06.568 INFO:scrape:
2024-01-13T07:03:06.568 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi148 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ec1182e55b4a38b17dd0d004d9fa6814d0269d4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py'
2024-01-13T07:03:06.568 INFO:scrape:1 jobs: ['7515353']
2024-01-13T07:03:06.568 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm_timeout}']
2024-01-13T07:03:06.568 INFO:scrape:
2024-01-13T07:03:06.573 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/
failed:  11
dead:    0
running: 0
waiting: 0
queued:  0
passed:  150


Fail
=================================================================
[7515083]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/1.7.2}
-----------------------------------------------------------------
time:   00:05:53
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515083/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515083/
sentry: https://sentry.ceph.com/organizations/ceph/?query=77e27f443ffb4903b52d48748e0f4ba5

    Command failed on smithi006 with status 100: "sudo apt update && sudo apt
    install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo
    /usr/share/keyrings/kubernetes-archive-keyring.gpg
    https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-
    by=/usr/share/keyrings/kubernetes-archive-keyring.gpg]
    https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee
    /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt
    install -y kubelet kubeadm kubectl bridge-utils"

[7515091]  orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:29:16
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515091/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515091/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9c437278f5a1456981f3ddcedc9b3fa0

    Command failed on smithi026 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    7c2afc9e-b1c6-11ee-95ac-87774f69a715 -e
    sha1=1ec1182e55b4a38b17dd0d004d9fa6814d0269d4 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7515122]  orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
-----------------------------------------------------------------
time:   00:11:51
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515122/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515122/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c9c41ead012c4a238fe63dcc2d102a3e

    Command failed on smithi149 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:1ec1182e55b4a38b17dd0d004d9fa6814d0269d4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    8a8d20ee-b1c8-11ee-95ac-87774f69a715 -- ceph mon dump -f json'

[7515164]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/host rook/master}
-----------------------------------------------------------------
time:   00:05:47
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515164/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515164/
sentry: https://sentry.ceph.com/organizations/ceph/?query=143c1e631ab144c2a7eb290c4451cb62

    Command failed on smithi186 with status 100: "sudo apt update && sudo apt
    install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo
    /usr/share/keyrings/kubernetes-archive-keyring.gpg
    https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-
    by=/usr/share/keyrings/kubernetes-archive-keyring.gpg]
    https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee
    /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt
    install -y kubelet kubeadm kubectl bridge-utils"

[7515244]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/calico rook/1.7.2}
-----------------------------------------------------------------
time:   00:05:45
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515244/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515244/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ab0781e9bd71498d9923440cacdb5402

    Command failed on smithi053 with status 100: "sudo apt update && sudo apt
    install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo
    /usr/share/keyrings/kubernetes-archive-keyring.gpg
    https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-
    by=/usr/share/keyrings/kubernetes-archive-keyring.gpg]
    https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee
    /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt
    install -y kubelet kubeadm kubectl bridge-utils"

[7515250]  orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:30:55
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515250/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515250/
sentry: https://sentry.ceph.com/organizations/ceph/?query=073660127f314494a189db8f4422b3f7

    Command failed on smithi086 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    1c2c6010-b1d2-11ee-95ac-87774f69a715 -e
    sha1=1ec1182e55b4a38b17dd0d004d9fa6814d0269d4 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7515325]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/master}
-----------------------------------------------------------------
time:   00:05:52
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515325/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515325/
sentry: https://sentry.ceph.com/organizations/ceph/?query=51441dd168cd40948f58d4382c50d79d

    Command failed on smithi012 with status 100: "sudo apt update && sudo apt
    install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo
    /usr/share/keyrings/kubernetes-archive-keyring.gpg
    https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-
    by=/usr/share/keyrings/kubernetes-archive-keyring.gpg]
    https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee
    /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt
    install -y kubelet kubeadm kubectl bridge-utils"

[7515343]  orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:08:59
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515343/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515343/
sentry: https://sentry.ceph.com/organizations/ceph/?query=72182984625749289cb70873b055dae5

    Command failed on smithi190 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    b6e0bdf8-b1d8-11ee-95ac-87774f69a715 -- ceph mon dump -f json'

[7515353]  orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_cephadm_timeout}
-----------------------------------------------------------------
time:   00:16:16
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515353/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515353/
sentry: https://sentry.ceph.com/organizations/ceph/?query=97bd91c381ac44bcaeb932f56efd9f3f

    Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi148
    with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd
    -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=1ec1182e55b4a38b17dd0d004d9fa6814d0269d4
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cephadm/test_cephadm_timeout.py'

[7515363]  orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:34:47
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515363/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515363/
sentry: https://sentry.ceph.com/organizations/ceph/?query=24f16c204d994f1aab607911f411656b

    Command failed (workunit test rados/test.sh) on smithi061 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=1ec1182e55b4a38b17dd0d004d9fa6814d0269d4
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

[7515385]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
-----------------------------------------------------------------
time:   00:48:14
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515385/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515385/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a87aed869ee442fd9e1af5dbfe33c59f

    Command failed on smithi073 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    a09901a2-b1da-11ee-95ac-87774f69a715 -e
    sha1=1ec1182e55b4a38b17dd0d004d9fa6814d0269d4 -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 7\'"\'"\'\''



Pass
=================================================================
[7515069] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_extra_daemon_features}
time:   00:14:56
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515069/

[7515071] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:17:59
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515071/

[7515073] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:29:49
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515073/

[7515075] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/iscsi 3-final}
time:   00:16:21
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515075/

[7515077] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_host_drain}
time:   00:16:10
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515077/

[7515079] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
time:   00:56:46
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515079/

[7515081] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:16:35
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515081/

[7515085] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/jaeger 3-final}
time:   00:23:07
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515085/

[7515087] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:12:54
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515087/

[7515089] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}}
time:   00:16:33
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515089/

[7515093] orch/cephadm/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs}
time:   00:42:53
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515093/

[7515095] orch/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0} 2-node-mgr agent/off orchestrator_cli}
time:   00:23:50
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515095/

[7515097] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi}
time:   00:30:27
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515097/

[7515099] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:13:23
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515099/

[7515101] orch/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 1-start 2-services/basic 3-final}
time:   00:13:39
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515101/

[7515103] orch/cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:11:36
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515103/

[7515105] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root}
time:   00:30:38
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515105/

[7515107] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
time:   00:51:54
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515107/

[7515108] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_monitoring_stack_basic}
time:   00:28:40
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515108/

[7515111] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final}
time:   00:13:36
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515111/

[7515113] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:30:57
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515113/

[7515115] orch/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:24:50
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515115/

[7515116] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
time:   00:19:36
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515116/

[7515118] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_orch_cli}
time:   00:18:30
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515118/

[7515120] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:12:49
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515120/

[7515124] orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root}
time:   00:46:51
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515124/

[7515126] orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
time:   00:28:56
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515126/

[7515128] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli_mon}
time:   00:24:33
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515128/

[7515130] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:23:05
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515130/

[7515132] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:29:38
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515132/

[7515134] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:14:13
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515134/

[7515136] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:15:16
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515136/

[7515138] orch/cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:11:18
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515138/

[7515140] orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic}
time:   00:44:00
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515140/

[7515142] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_rgw_multisite}
time:   00:16:36
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515142/

[7515144] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:17:54
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515144/

[7515146] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
time:   00:13:26
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515146/

[7515148] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_set_mon_crush_locations}
time:   00:26:13
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515148/

[7515150] orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root}
time:   00:57:32
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515150/

[7515152] orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys}
time:   00:31:30
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515152/

[7515154] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:20:40
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515154/

[7515156] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_adoption}
time:   00:18:32
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515156/

[7515158] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:13:40
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515158/

[7515160] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:15:32
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515160/

[7515162] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:30:02
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515162/

[7515166] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:15:49
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515166/

[7515168] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nvmeof 3-final}
time:   00:18:19
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515168/

[7515170] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_ca_signed_key}
time:   00:15:52
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515170/

[7515172] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/on orchestrator_cli}
time:   00:15:47
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515172/

[7515174] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:15:34
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515174/

[7515176] orch/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 1-start 2-services/rgw 3-final}
time:   00:14:27
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515176/

[7515178] orch/cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:11:37
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515178/

[7515180] orch/cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
time:   01:12:20
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515180/

[7515182] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests}
time:   00:59:22
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515182/

[7515184] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_cephadm}
time:   00:16:33
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515184/

[7515186] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:20:03
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515186/

[7515188] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:14:54
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515188/

[7515190] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
time:   00:43:15
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515190/

[7515192] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_cephadm_repos}
time:   00:07:49
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515192/

[7515194] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:15:12
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515194/

[7515196] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final}
time:   00:17:40
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515196/

[7515198] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:30:11
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515198/

[7515200] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:16:04
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515200/

[7515202] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_cephadm_timeout}
time:   00:23:16
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515202/

[7515204] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root}
time:   00:29:31
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515204/

[7515206] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python}
time:   00:22:05
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515206/

[7515208] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:16:34
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515208/

[7515210] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/iscsi 3-final}
time:   00:18:38
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515210/

[7515212] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_extra_daemon_features}
time:   00:21:39
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515212/

[7515214] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/jaeger 3-final}
time:   00:13:43
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515214/

[7515216] orch/cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:11:56
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515216/

[7515218] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:18:38
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515218/

[7515220] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:13:04
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515220/

[7515222] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_host_drain}
time:   00:22:01
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515222/

[7515224] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:30:52
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515224/

[7515226] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root}
time:   00:36:30
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515226/

[7515228] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys}
time:   00:26:53
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515228/

[7515230] orch/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:24:07
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515230/

[7515232] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final}
time:   00:20:16
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515232/

[7515234] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}}
time:   00:18:33
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515234/

[7515236] orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic}
time:   00:45:06
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515236/

[7515238] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
time:   00:17:53
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515238/

[7515240] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:12:36
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515240/

[7515242] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_monitoring_stack_basic}
time:   00:22:57
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515242/

[7515245] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:21:26
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515245/

[7515248] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:31:46
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515248/

[7515252] orch/cephadm/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs}
time:   00:32:32
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515252/

[7515253] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/off orchestrator_cli}
time:   00:18:14
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515253/

[7515256] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi}
time:   00:35:02
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515256/

[7515258] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:13:48
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515258/

[7515260] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:15:54
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515260/

[7515261] orch/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 1-start 2-services/basic 3-final}
time:   00:14:56
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515261/

[7515263] orch/cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:11:10
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515263/

[7515265] orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root}
time:   00:45:04
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515265/

[7515267] orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
time:   01:03:02
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515267/

[7515269] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_orch_cli}
time:   00:25:15
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515269/

[7515271] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:15:20
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515271/

[7515273] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:12:28
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515273/

[7515275] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_orch_cli_mon}
time:   00:31:28
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515275/

[7515277] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
time:   00:18:48
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515277/

[7515279] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:14:02
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515279/

[7515281] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:16:39
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515281/

[7515283] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_rgw_multisite}
time:   00:18:54
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515283/

[7515285] orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root}
time:   00:49:25
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515285/

[7515287] orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
time:   00:26:26
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515287/

[7515289] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:30:06
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515289/

[7515291] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
time:   00:37:40
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515291/

[7515293] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:18:50
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515293/

[7515295] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_set_mon_crush_locations}
time:   00:17:37
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515295/

[7515297] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:19:12
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515297/

[7515299] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nvmeof 3-final}
time:   00:12:56
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515299/

[7515301] orch/cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:11:29
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515301/

[7515303] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_adoption}
time:   00:12:10
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515303/

[7515305] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:18:51
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515305/

[7515307] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:15:13
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515307/

[7515309] orch/cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root}
time:   00:43:26
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515309/

[7515311] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys}
time:   00:37:41
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515311/

[7515313] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:20:03
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515313/

[7515315] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_ca_signed_key}
time:   00:21:42
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515315/

[7515317] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:32:00
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515317/

[7515319] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:17:30
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515319/

[7515321] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/basic 3-final}
time:   00:16:21
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515321/

[7515323] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_cephadm}
time:   00:24:41
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515323/

[7515327] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:16:56
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515327/

[7515329] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:19:22
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515329/

[7515331] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_cephadm_repos}
time:   00:07:09
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515331/

[7515333] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/on orchestrator_cli}
time:   00:15:24
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515333/

[7515335] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/iscsi 3-final}
time:   00:12:46
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515335/

[7515337] orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final}
time:   00:18:43
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515337/

[7515339] orch/cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:11:32
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515339/

[7515341] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root}
time:   00:35:42
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515341/

[7515345] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests}
time:   00:52:11
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515345/

[7515347] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/jaeger 3-final}
time:   00:13:19
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515347/

[7515349] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:15:08
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515349/

[7515351] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:30:22
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515351/

[7515355] orch/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:24:12
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515355/

[7515357] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:19:50
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515357/

[7515359] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_extra_daemon_features}
time:   00:15:27
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515359/

[7515361] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final}
time:   00:17:45
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515361/

[7515365] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python}
time:   00:22:15
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515365/

[7515367] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
time:   00:20:54
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515367/

[7515369] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_host_drain}
time:   00:25:51
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515369/

[7515371] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:18:42
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515371/

[7515373] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:29:53
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515373/

[7515375] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:13:44
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515375/

[7515377] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:14:28
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515377/

[7515379] orch/cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:11:25
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515379/

[7515381] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}}
time:   00:14:54
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515381/

[7515383] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:14:45
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515383/

[7515387] orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root}
time:   00:57:04
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:25:05-orch-main-distro-default-smithi/7515387/