2022-10-01T10:38:32.176 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-01T10:38:32.177 INFO:teuthology.results:subset: '111/120000'
2022-10-01T10:38:32.178 INFO:teuthology.results:seed: '2841'
2022-10-01T12:45:36.259 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-01T12:45:36.285 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-10-01T13:25:39.815 INFO:teuthology.results:Tests finished! gathering results...
2022-10-01T13:25:49.520 INFO:scrape:Found 137 jobs
2022-10-01T13:25:54.390 INFO:scrape:Found 8 distinct failure reasons
2022-10-01T13:25:54.391 INFO:scrape:Failure: Command failed on smithi155 with status 2: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9885d166-4188-11ed-8432-001a4aab830c -- ceph-volume lvm zap /dev/nvme1n1'
2022-10-01T13:25:54.392 INFO:scrape:2 jobs: ['7049976', '7049931']
2022-10-01T13:25:54.392 INFO:scrape:suites intersection: ['0-nvme-loop', 'agent/on', 'fixed-2', 'mon_election/connectivity', 'start}']
2022-10-01T13:25:54.393 INFO:scrape:suites union: ['0-nvme-loop', 'agent/on', 'fixed-2', 'mon_election/connectivity', 'orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'start}']
2022-10-01T13:25:54.393 INFO:scrape:
2022-10-01T13:25:54.394 INFO:scrape:Failure: 'wait for operator' reached maximum tries (90) after waiting for 900 seconds
2022-10-01T13:25:54.394 INFO:scrape:2 jobs: ['7049861', '7049930']
2022-10-01T13:25:54.394 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}']
2022-10-01T13:25:54.395 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/flannel', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}']
2022-10-01T13:25:54.395 INFO:scrape:
2022-10-01T13:25:54.396 INFO:scrape:Failure: Command failed on smithi196 with status 1: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2022-10-01T13:25:54.396 INFO:scrape:1 jobs: ['7049977']
2022-10-01T13:25:54.397 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'task/test_iscsi_pids_limit/{centos_8.stream_container_tools', 'test_iscsi_pids_limit}}']
2022-10-01T13:25:54.397 INFO:scrape:
2022-10-01T13:25:54.397 INFO:scrape:Failure: Command failed on smithi093 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 28f7420e-4187-11ed-8432-001a4aab830c -e sha1=7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''
2022-10-01T13:25:54.398 INFO:scrape:10 jobs: ['7049972', '7049851', '7049937', '7049878', '7049884', '7049905', '7049919', '7049946', '7049869', '7049954']
2022-10-01T13:25:54.398 INFO:scrape:suites intersection: []
2022-10-01T13:25:54.399 INFO:scrape:suites union: ['1-bootstrap/octopus', '1-start', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9']
2022-10-01T13:25:54.399 INFO:scrape:
2022-10-01T13:25:54.400 INFO:scrape:Failure: Config file not found: "/home/teuthworker/src/github.com_ceph_ceph-c_c667ad96af9106b9ae2e23649842d2561ec265bf/qa/tasks/cephadm.conf".
2022-10-01T13:25:54.400 INFO:scrape:1 jobs: ['7049912']
2022-10-01T13:25:54.401 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs', '3-final}', 'orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools']
2022-10-01T13:25:54.401 INFO:scrape:
2022-10-01T13:25:54.402 INFO:scrape:Failure: No module named 'tasks'
2022-10-01T13:25:54.402 INFO:scrape:1 jobs: ['7049924']
2022-10-01T13:25:54.403 INFO:scrape:suites: ['1-start', '2-thrash', '3-tasks/small-objects', 'fixed-2', 'msgr/async', 'orch/cephadm/thrash/{0-distro/ubuntu_20.04', 'root}']
2022-10-01T13:25:54.403 INFO:scrape:
2022-10-01T13:25:54.403 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi035.front.sepia.ceph.com: ['type=AVC msg=audit(1664625037.836:19062): avc:  denied  { ioctl } for  pid=112085 comm="iptables" path="/var/lib/containers/storage/overlay/7770def4d05ca47577fa3f66cbf08d56548f5a80e4e75c0e13354f1c1904bbf9/merged" dev="overlay" ino=3409979 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1']
2022-10-01T13:25:54.404 INFO:scrape:1 jobs: ['7049895']
2022-10-01T13:25:54.404 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_cephadm}']
2022-10-01T13:25:54.405 INFO:scrape:
2022-10-01T13:25:54.405 INFO:scrape:Failure: Command failed on smithi066 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml'
2022-10-01T13:25:54.406 INFO:scrape:2 jobs: ['7049896', '7049965']
2022-10-01T13:25:54.406 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/radosbench', 'cluster/1-node', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}']
2022-10-01T13:25:54.406 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/radosbench', 'cluster/1-node', 'k8s/1.21', 'net/calico', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}']
2022-10-01T13:25:54.407 INFO:scrape:
2022-10-01T13:25:54.412 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/
failed:  20
dead:    0
running: 0
waiting: 0
queued:  0
passed:  117


Fail
=================================================================
[7049851]  orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
-----------------------------------------------------------------
time:   00:18:32
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049851/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049851/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5e1a96aa9e16442d81a13f71f413788b

    Command failed on smithi006 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    776fad32-4177-11ed-8432-001a4aab830c -e
    sha1=7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7049861]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/host rook/master}
-----------------------------------------------------------------
time:   00:21:45
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049861/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049861/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e77a49cbc1c3415e8c7dee0f5ce2396c

    'wait for operator' reached maximum tries (90) after waiting for 900
    seconds

[7049869]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic}
-----------------------------------------------------------------
time:   00:52:26
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049869/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049869/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5ac72c88730d47ab885af43bc8edca4a

    Command failed on smithi083 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.9 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    978fbd1c-4179-11ed-8432-001a4aab830c -e
    sha1=7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 2\'"\'"\'\''

[7049878]  orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   01:00:31
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049878/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049878/
sentry: https://sentry.ceph.com/organizations/ceph/?query=130d78872ce34ca7b845285fa4a71312

    Command failed on smithi006 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    f4a1e40c-417a-11ed-8432-001a4aab830c -e
    sha1=7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7049884]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
-----------------------------------------------------------------
time:   00:20:19
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049884/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049884/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1e337ae30235486e94743b3f250e9ab1

    Command failed on smithi120 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    3984c3e0-417c-11ed-8432-001a4aab830c -e
    sha1=7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7049895]  orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm}
-----------------------------------------------------------------
time:   00:19:40
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049895/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049895/

    SELinux denials found on ubuntu@smithi035.front.sepia.ceph.com: ['type=AVC
    msg=audit(1664625037.836:19062): avc:  denied  { ioctl } for  pid=112085
    comm="iptables" path="/var/lib/containers/storage/overlay/7770def4d05ca4757
    7fa3f66cbf08d56548f5a80e4e75c0e13354f1c1904bbf9/merged" dev="overlay"
    ino=3409979 scontext=system_u:system_r:iptables_t:s0
    tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir
    permissive=1']

[7049896]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/calico rook/1.7.2}
-----------------------------------------------------------------
time:   00:06:23
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049896/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049896/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2271f422e5c641d8a2480009f04a5c1b

    Command failed on smithi066 with status 1: 'kubectl create -f
    rook/cluster/examples/kubernetes/ceph/crds.yaml -f
    rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml'

[7049905]  orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:18:41
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049905/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049905/
sentry: https://sentry.ceph.com/organizations/ceph/?query=597b0bf6efe0470e82ab1b16d142002a

    Command failed on smithi008 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    4eb672ce-417f-11ed-8432-001a4aab830c -e
    sha1=7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7049912]  orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs 3-final}
-----------------------------------------------------------------
time:   00:07:17
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049912/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049912/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6e73a53bdc20479c998c2ea4b3dddafa

    Config file not found: "/home/teuthworker/src/github.com_ceph_ceph-
    c_c667ad96af9106b9ae2e23649842d2561ec265bf/qa/tasks/cephadm.conf".

[7049919]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
-----------------------------------------------------------------
time:   00:52:19
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049919/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049919/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dc2b4505aeed486cb605996bc709cf1b

    Command failed on smithi037 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.9 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    099b2f34-4181-11ed-8432-001a4aab830c -e
    sha1=7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 2\'"\'"\'\''

[7049924]  orch/cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:06:30
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049924/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049924/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b2de5895deaa4faeb790d0cc4497605f

    No module named 'tasks'

[7049930]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/master}
-----------------------------------------------------------------
time:   00:27:08
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049930/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049930/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5ca95e62cc2a4eadb30548a1a6c9cd7b

    'check osd count' reached maximum tries (90) after waiting for 900 seconds

[7049931]  orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:10:59
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049931/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049931/
sentry: https://sentry.ceph.com/organizations/ceph/?query=72eef59f664b4ef98cdfb032cff0657c

    Command failed on smithi191 with status 127: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    c1e13844-4182-11ed-8432-001a4aab830c -- ceph mon dump -f json'

[7049937]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:20:20
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049937/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049937/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e1485c45a59e402ca5f9890db6d6b0e7

    Command failed on smithi112 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    43b93664-4183-11ed-8432-001a4aab830c -e
    sha1=7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7049946]  orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:59:55
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049946/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049946/
sentry: https://sentry.ceph.com/organizations/ceph/?query=95b376b7a5cd43a68c4a0a3658070457

    Command failed on smithi043 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    6b477ff0-4184-11ed-8432-001a4aab830c -e
    sha1=7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7049954]  orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
-----------------------------------------------------------------
time:   00:50:13
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049954/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049954/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e58506f164e44ad8832ce8fcd62673c7

    Command failed on smithi078 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    5c8ae8ac-4185-11ed-8432-001a4aab830c -e
    sha1=7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 2\'"\'"\'\''

[7049965]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/host rook/1.7.2}
-----------------------------------------------------------------
time:   00:08:17
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049965/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049965/
sentry: https://sentry.ceph.com/organizations/ceph/?query=74025d56df344fc4aa5542597daae0fc

    Command failed on smithi033 with status 1: 'kubectl create -f
    rook/cluster/examples/kubernetes/ceph/crds.yaml -f
    rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml'

[7049972]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:20:15
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049972/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049972/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5a40a2002a5047748d0289ba1d23631e

    Command failed on smithi093 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.9 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    28f7420e-4187-11ed-8432-001a4aab830c -e
    sha1=7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7049976]  orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:16:36
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049976/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049976/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1a29665b84f44647a0489102ec9133d2

    Command failed on smithi155 with status 2: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:7e3a743186065a6e76e2eff2954ddd8fb2bcf8cd shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    9885d166-4188-11ed-8432-001a4aab830c -- ceph-volume lvm zap /dev/nvme1n1'

[7049977]  orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_iscsi_pids_limit/{centos_8.stream_container_tools test_iscsi_pids_limit}}
-----------------------------------------------------------------
time:   00:06:05
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049977/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049977/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fb49265177684e929713993caafc453e

    Command failed on smithi196 with status 1: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'



Pass
=================================================================
[7049843] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:14:22
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049843/

[7049844] orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root}
time:   00:41:23
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049844/

[7049845] orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
time:   00:28:35
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049845/

[7049846] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:19:58
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049846/

[7049847] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:17:11
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049847/

[7049848] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_cephadm_repos}
time:   00:10:01
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049848/

[7049849] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:31:17
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049849/

[7049850] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:16:57
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049850/

[7049852] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:17:24
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049852/

[7049853] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:17:17
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049853/

[7049854] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:32:25
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049854/

[7049855] orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root}
time:   00:38:53
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049855/

[7049856] orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys}
time:   00:28:26
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049856/

[7049857] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:16:41
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049857/

[7049858] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_iscsi_pids_limit/{centos_8.stream_container_tools test_iscsi_pids_limit}}
time:   00:15:31
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049858/

[7049859] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:16:51
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049859/

[7049860] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:16:27
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049860/

[7049862] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:32:26
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049862/

[7049863] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:16:33
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049863/

[7049864] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:23:58
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049864/

[7049865] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_nfs}
time:   00:31:13
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049865/

[7049866] orch/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0} 2-node-mgr agent/on orchestrator_cli}
time:   00:18:23
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049866/

[7049867] orch/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/rgw 3-final}
time:   00:16:09
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049867/

[7049868] orch/cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
time:   01:03:39
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049868/

[7049870] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests}
time:   00:43:31
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049870/

[7049871] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:15:57
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049871/

[7049872] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:19:38
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049872/

[7049873] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:15:57
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049873/

[7049874] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/basic 3-final}
time:   00:13:15
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049874/

[7049875] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli}
time:   00:15:36
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049875/

[7049876] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:31:08
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049876/

[7049877] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:17:42
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049877/

[7049879] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root}
time:   00:32:49
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049879/

[7049880] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python}
time:   00:24:35
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049880/

[7049881] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:18:46
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049881/

[7049882] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_orch_cli_mon}
time:   00:25:34
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049882/

[7049883] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/iscsi 3-final}
time:   00:16:52
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049883/

[7049885] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:17:29
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049885/

[7049886] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/jaeger 3-final}
time:   00:19:07
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049886/

[7049887] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:31:16
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049887/

[7049888] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root}
time:   00:37:15
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049888/

[7049889] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys}
time:   00:25:52
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049889/

[7049890] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_adoption}
time:   00:14:11
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049890/

[7049891] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:14:53
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049891/

[7049892] orch/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:24:11
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049892/

[7049893] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
time:   00:15:44
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049893/

[7049894] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:14:12
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049894/

[7049897] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:18:37
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049897/

[7049898] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:31:48
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049898/

[7049899] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:36:57
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049899/

[7049900] orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_20.04} 2-node-mgr agent/off orchestrator_cli}
time:   00:13:04
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049900/

[7049901] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi}
time:   00:32:07
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049901/

[7049902] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:16:49
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049902/

[7049903] orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final}
time:   00:16:17
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049903/

[7049904] orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root}
time:   00:39:46
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049904/

[7049906] orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
time:   00:39:33
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049906/

[7049907] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:19:06
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049907/

[7049908] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:13:36
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049908/

[7049909] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm_repos}
time:   00:05:43
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049909/

[7049910] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:22:32
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049910/

[7049911] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:16:56
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049911/

[7049913] orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root}
time:   00:56:36
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049913/

[7049914] orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
time:   00:26:49
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049914/

[7049915] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:31:26
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049915/

[7049916] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_iscsi_pids_limit/{centos_8.stream_container_tools test_iscsi_pids_limit}}
time:   00:14:54
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049916/

[7049917] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:14:33
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049917/

[7049918] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:18:14
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049918/

[7049920] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:20:59
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049920/

[7049921] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:19:43
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049921/

[7049922] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_nfs}
time:   00:29:26
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049922/

[7049923] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:32:47
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049923/

[7049925] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys}
time:   00:38:03
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049925/

[7049926] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:18:17
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049926/

[7049927] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:32:42
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049927/

[7049928] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:15:57
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049928/

[7049929] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/basic 3-final}
time:   00:16:18
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049929/

[7049932] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_orch_cli}
time:   00:18:58
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049932/

[7049933] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:14:19
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049933/

[7049934] orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_20.04} 2-node-mgr agent/on orchestrator_cli}
time:   00:13:05
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049934/

[7049935] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/rgw 3-final}
time:   00:13:37
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049935/

[7049936] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root}
time:   00:38:01
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049936/

[7049938] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests}
time:   00:37:25
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049938/

[7049939] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/iscsi 3-final}
time:   00:14:09
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049939/

[7049940] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:16:41
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049940/

[7049941] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:31:17
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049941/

[7049942] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli_mon}
time:   00:29:05
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049942/

[7049943] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/jaeger 3-final}
time:   00:17:29
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049943/

[7049944] orch/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:25:06
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049944/

[7049945] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:16:49
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049945/

[7049947] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root}
time:   00:37:55
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049947/

[7049948] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python}
time:   00:24:16
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049948/

[7049949] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_adoption}
time:   00:09:08
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049949/

[7049950] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
time:   00:19:49
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049950/

[7049951] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:18:09
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049951/

[7049952] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:31:52
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049952/

[7049953] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:16:53
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049953/

[7049955] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:15:59
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049955/

[7049956] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_cephadm}
time:   00:16:36
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049956/

[7049957] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:16:47
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049957/

[7049958] orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root}
time:   00:55:00
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049958/

[7049959] orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys}
time:   00:28:08
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049959/

[7049960] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:14:03
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049960/

[7049961] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:19:49
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049961/

[7049962] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:16:31
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049962/

[7049963] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_cephadm_repos}
time:   00:07:36
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049963/

[7049964] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:31:39
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049964/

[7049966] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:16:09
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049966/

[7049967] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:37:00
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049967/

[7049968] orch/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 2-node-mgr agent/off orchestrator_cli}
time:   00:19:20
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049968/

[7049969] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi}
time:   00:40:38
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049969/

[7049970] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/basic 3-final}
time:   00:12:32
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049970/

[7049971] orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root}
time:   00:35:43
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049971/

[7049973] orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
time:   00:40:23
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049973/

[7049974] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:17:25
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049974/

[7049975] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:14:34
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049975/

[7049978] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:18:36
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049978/

[7049979] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:31:04
info:   http://pulpito.ceph.com/yuriw-2022-09-30_17:56:14-orch-wip-all-kickoff-r-distro-default-smithi/7049979/