2022-10-14T19:21:01.321 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-14T19:21:01.323 INFO:teuthology.results:subset: '111/120000'
2022-10-14T19:21:01.323 INFO:teuthology.results:seed: '4563'
2022-10-14T20:44:47.454 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-14T20:44:47.476 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-10-14T21:28:51.131 INFO:teuthology.results:Tests finished! gathering results...
2022-10-14T21:29:00.178 INFO:scrape:Found 132 jobs
2022-10-14T21:29:04.425 INFO:scrape:Found 5 distinct failure reasons
2022-10-14T21:29:04.427 INFO:scrape:Failure: Command failed on smithi088 with status 1: 'sudo yum -y install ceph-radosgw'
2022-10-14T21:29:04.427 INFO:scrape:18 jobs: ['7066031', '7066143', '7066145', '7066088', '7066065', '7066070', '7066032', '7066132', '7066075', '7066149', '7066079', '7066021', '7066115', '7066089', '7066022', '7066081', '7066106', '7066131']
2022-10-14T21:29:04.428 INFO:scrape:suites intersection: []
2022-10-14T21:29:04.428 INFO:scrape:suites union: ['1-start', '2-node-mgr', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'agent/off', 'agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0', 'orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8', 'orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'orchestrator_cli}', 'root}', 'start', 'task/test_adoption}', 'task/test_cephadm}', 'task/test_iscsi_pids_limit/{centos_8.stream_container_tools', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'test_iscsi_pids_limit}}']
2022-10-14T21:29:04.429 INFO:scrape:
2022-10-14T21:29:04.429 INFO:scrape:Failure: Command failed on smithi026 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41b8586e-4bfa-11ed-8437-001a4aab830c -e sha1=6fb859258928a5d76fd5f6b6802b10593037f421 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''
2022-10-14T21:29:04.430 INFO:scrape:11 jobs: ['7066059', '7066119', '7066093', '7066144', '7066127', '7066027', '7066053', '7066045', '7066080', '7066111', '7066148']
2022-10-14T21:29:04.430 INFO:scrape:suites intersection: []
2022-10-14T21:29:04.431 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/octopus', '1-start', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9', 'start}']
2022-10-14T21:29:04.431 INFO:scrape:
2022-10-14T21:29:04.432 INFO:scrape:Failure: 'wait for operator' reached maximum tries (90) after waiting for 900 seconds
2022-10-14T21:29:04.432 INFO:scrape:2 jobs: ['7066037', '7066104']
2022-10-14T21:29:04.433 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}']
2022-10-14T21:29:04.433 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/flannel', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}']
2022-10-14T21:29:04.433 INFO:scrape:
2022-10-14T21:29:04.434 INFO:scrape:Failure: Command failed on smithi142 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml'
2022-10-14T21:29:04.434 INFO:scrape:2 jobs: ['7066071', '7066137']
2022-10-14T21:29:04.435 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/radosbench', 'cluster/1-node', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}']
2022-10-14T21:29:04.435 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/radosbench', 'cluster/1-node', 'k8s/1.21', 'net/calico', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}']
2022-10-14T21:29:04.436 INFO:scrape:
2022-10-14T21:29:04.436 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi137.front.sepia.ceph.com: ['type=AVC msg=audit(1665780597.013:7030): avc:  denied  { ioctl } for  pid=58817 comm="iptables" path="/var/lib/containers/storage/overlay/4de28fee34d8841439fc6bde7fea232f2087adc9a3bdf14588dc071097fb95cd/merged" dev="overlay" ino=3281439 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1']
2022-10-14T21:29:04.437 INFO:scrape:1 jobs: ['7066129']
2022-10-14T21:29:04.437 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}']
2022-10-14T21:29:04.437 INFO:scrape:
2022-10-14T21:29:04.447 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/
failed:  34
dead:    0
running: 0
waiting: 0
queued:  0
passed:  98


Fail
=================================================================
[7066021]  orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:08:08
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066021/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066021/
sentry: https://sentry.ceph.com/organizations/ceph/?query=375a17ea02294128906f3bd05c99bef3

    Command failed on smithi150 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066022]  orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:08:38
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066022/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066022/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b13641be55a04221965af8102b10de92

    Command failed on smithi005 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066027]  orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
-----------------------------------------------------------------
time:   00:16:24
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066027/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066027/
sentry: https://sentry.ceph.com/organizations/ceph/?query=45ab36ef4bfa4c358bddf7bf674bbb08

    Command failed on smithi032 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    b70d4f64-4bf7-11ed-8437-001a4aab830c -e
    sha1=6fb859258928a5d76fd5f6b6802b10593037f421 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7066031]  orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:08:42
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066031/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066031/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d07a7f675bd143d39b63e7766bb7e3d6

    Command failed on smithi088 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066032]  orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:08:47
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066032/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066032/
sentry: https://sentry.ceph.com/organizations/ceph/?query=df45ca3948924a8ba1c2441d590b8ca8

    Command failed on smithi154 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066037]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/host rook/master}
-----------------------------------------------------------------
time:   00:19:35
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066037/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066037/
sentry: https://sentry.ceph.com/organizations/ceph/?query=179e5f28013e4f79b06993dddef69941

    'wait for operator' reached maximum tries (90) after waiting for 900
    seconds

[7066045]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic}
-----------------------------------------------------------------
time:   00:51:11
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066045/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066045/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aeefcac33c5b4ecda27a23cf104597b0

    Command failed on smithi006 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.9 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    c2e0d418-4bf8-11ed-8437-001a4aab830c -e
    sha1=6fb859258928a5d76fd5f6b6802b10593037f421 -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 2\'"\'"\'\''

[7066053]  orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:58:12
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066053/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066053/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ed105a33110f4f35a2390d0417b83a90

    Command failed on smithi124 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    40fd5ee8-4bf9-11ed-8437-001a4aab830c -e
    sha1=6fb859258928a5d76fd5f6b6802b10593037f421 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7066059]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
-----------------------------------------------------------------
time:   00:18:39
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066059/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066059/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9b83a7d460fa4d458f968c40febb8d4d

    Command failed on smithi026 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    41b8586e-4bfa-11ed-8437-001a4aab830c -e
    sha1=6fb859258928a5d76fd5f6b6802b10593037f421 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7066065]  orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_adoption}
-----------------------------------------------------------------
time:   00:07:59
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066065/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066065/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7e24d129d94944ca92793a7937e0d4d5

    Command failed on smithi083 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066070]  orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm}
-----------------------------------------------------------------
time:   00:08:27
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066070/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066070/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f6c75a1e0803426aac9fb0163c4ec80d

    Command failed on smithi040 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066071]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/calico rook/1.7.2}
-----------------------------------------------------------------
time:   00:04:28
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066071/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066071/
sentry: https://sentry.ceph.com/organizations/ceph/?query=342bc2b87e784ed9b4d27a4be1949180

    Command failed on smithi142 with status 1: 'kubectl create -f
    rook/cluster/examples/kubernetes/ceph/crds.yaml -f
    rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml'

[7066075]  orch/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0} 2-node-mgr agent/off orchestrator_cli}
-----------------------------------------------------------------
time:   00:07:59
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066075/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066075/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c08e9f09aade4a5daee7a7abe33285bd

    Command failed on smithi157 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066079]  orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:07:53
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066079/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066079/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8f5595a3ab374ea588f4144b23ced5b0

    Command failed on smithi156 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066080]  orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:16:18
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066080/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066080/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a7682a7497c94dfd915f831b6566a2a5

    Command failed on smithi079 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    f27dffd6-4bfb-11ed-8437-001a4aab830c -e
    sha1=6fb859258928a5d76fd5f6b6802b10593037f421 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7066081]  orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:07:58
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066081/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066081/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b04887c346df40c89ebbe9ff127d2787

    Command failed on smithi196 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066088]  orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:08:16
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066088/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066088/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3aa1a502d09840ca9fa187b2e4e03df7

    Command failed on smithi155 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066089]  orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:08:17
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066089/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066089/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4f431c6cbb9f4ab29b62966b10c6810d

    Command failed on smithi190 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066093]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
-----------------------------------------------------------------
time:   00:50:44
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066093/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066093/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9c17a219a2414e4195e7e1d2549a7582

    Command failed on smithi157 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.9 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    ec3ee2dc-4bfd-11ed-8437-001a4aab830c -e
    sha1=6fb859258928a5d76fd5f6b6802b10593037f421 -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 2\'"\'"\'\''

[7066104]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/master}
-----------------------------------------------------------------
time:   00:25:44
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066104/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066104/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ec24cd09a4b34ed49fc6f0bcaaa441b3

    'check osd count' reached maximum tries (90) after waiting for 900 seconds

[7066106]  orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_orch_cli}
-----------------------------------------------------------------
time:   00:07:58
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066106/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066106/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2e41ad5d2a84415abbec831e964669ea

    Command failed on smithi137 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066111]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:19:04
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066111/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066111/
sentry: https://sentry.ceph.com/organizations/ceph/?query=56a2842ab9f44f46b06a308a72409645

    Command failed on smithi045 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    e061dd10-4bfe-11ed-8437-001a4aab830c -e
    sha1=6fb859258928a5d76fd5f6b6802b10593037f421 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7066115]  orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli_mon}
-----------------------------------------------------------------
time:   00:08:27
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066115/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066115/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9d7959deee384f66b015f7d3c297b16f

    Command failed on smithi190 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066119]  orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:58:14
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066119/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066119/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b44bad975c584d19b09b25f8475b1b77

    Command failed on smithi047 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    6d59d678-4bff-11ed-8437-001a4aab830c -e
    sha1=6fb859258928a5d76fd5f6b6802b10593037f421 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7066127]  orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
-----------------------------------------------------------------
time:   00:47:43
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066127/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066127/
sentry: https://sentry.ceph.com/organizations/ceph/?query=007a4580fdea46c19763aee1f441ba79

    Command failed on smithi050 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    3533e8fa-4c00-11ed-8437-001a4aab830c -e
    sha1=6fb859258928a5d76fd5f6b6802b10593037f421 -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 2\'"\'"\'\''

[7066129]  orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_cephadm}
-----------------------------------------------------------------
time:   00:13:25
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066129/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066129/

    SELinux denials found on ubuntu@smithi137.front.sepia.ceph.com: ['type=AVC
    msg=audit(1665780597.013:7030): avc:  denied  { ioctl } for  pid=58817
    comm="iptables" path="/var/lib/containers/storage/overlay/4de28fee34d884143
    9fc6bde7fea232f2087adc9a3bdf14588dc071097fb95cd/merged" dev="overlay"
    ino=3281439 scontext=system_u:system_r:iptables_t:s0
    tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir
    permissive=1']

[7066131]  orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:07:54
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066131/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066131/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dd33b819d05c41018fba86e23b4888f4

    Command failed on smithi182 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066132]  orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:07:56
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066132/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066132/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c8c643d90b6946b8b468728d8ca32187

    Command failed on smithi143 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066137]  orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/host rook/1.7.2}
-----------------------------------------------------------------
time:   00:04:34
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066137/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066137/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a464ddd20e9f4d159c5732331c77bbc3

    Command failed on smithi017 with status 1: 'kubectl create -f
    rook/cluster/examples/kubernetes/ceph/crds.yaml -f
    rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml'

[7066143]  orch/cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:08:27
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066143/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066143/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0e1d63ac78a94d888a283f2027044303

    Command failed on smithi033 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066144]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:18:32
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066144/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066144/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9e53f701efc9479eb225a4df182638e1

    Command failed on smithi189 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v15.2.9 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    9cdb85f2-4c01-11ed-8437-001a4aab830c -e
    sha1=6fb859258928a5d76fd5f6b6802b10593037f421 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7066145]  orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:08:07
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066145/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066145/
sentry: https://sentry.ceph.com/organizations/ceph/?query=35489556ccf84ea4914e3386a8ccab0f

    Command failed on smithi156 with status 1: 'sudo yum -y install ceph-
    radosgw'

[7066148]  orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:10:42
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066148/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066148/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2ddf8db00e874cdcb0da8abd442f610e

    Command failed on smithi052 with status 127: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:6fb859258928a5d76fd5f6b6802b10593037f421 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    3f20ca2a-4c02-11ed-8437-001a4aab830c -- ceph mon dump -f json'

[7066149]  orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_iscsi_pids_limit/{centos_8.stream_container_tools test_iscsi_pids_limit}}
-----------------------------------------------------------------
time:   00:03:07
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066149/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066149/
sentry: https://sentry.ceph.com/organizations/ceph/?query=70c0a6dcd15041219674ecb70904641f

    Command failed on smithi111 with status 1: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'



Pass
=================================================================
[7066020] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:10:59
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066020/

[7066023] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:19:23
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066023/

[7066024] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:14:28
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066024/

[7066025] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_cephadm_repos}
time:   00:09:31
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066025/

[7066026] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:13:14
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066026/

[7066028] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:16:40
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066028/

[7066029] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:13:52
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066029/

[7066030] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:31:10
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066030/

[7066033] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:16:39
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066033/

[7066034] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_iscsi_pids_limit/{centos_8.stream_container_tools test_iscsi_pids_limit}}
time:   00:12:54
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066034/

[7066035] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:13:33
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066035/

[7066036] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:16:28
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066036/

[7066038] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:30:09
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066038/

[7066039] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:16:55
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066039/

[7066040] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:23:06
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066040/

[7066041] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_nfs}
time:   00:30:49
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066041/

[7066042] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/on orchestrator_cli}
time:   00:13:00
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066042/

[7066043] orch/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/rgw 3-final}
time:   00:15:45
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066043/

[7066044] orch/cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
time:   01:01:54
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066044/

[7066046] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests}
time:   00:41:09
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066046/

[7066047] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:12:37
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066047/

[7066048] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:17:47
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066048/

[7066049] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:14:30
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066049/

[7066050] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/basic 3-final}
time:   00:10:38
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066050/

[7066051] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli}
time:   00:12:31
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066051/

[7066052] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:16:26
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066052/

[7066054] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root}
time:   00:29:20
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066054/

[7066055] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python}
time:   00:21:32
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066055/

[7066056] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:18:45
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066056/

[7066057] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_orch_cli_mon}
time:   00:22:04
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066057/

[7066058] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/iscsi 3-final}
time:   00:15:31
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066058/

[7066060] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:16:16
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066060/

[7066061] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/jaeger 3-final}
time:   00:17:00
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066061/

[7066062] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:29:36
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066062/

[7066063] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root}
time:   00:34:51
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066063/

[7066064] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys}
time:   00:22:17
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066064/

[7066066] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:11:36
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066066/

[7066067] orch/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:22:08
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066067/

[7066068] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
time:   00:13:07
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066068/

[7066069] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:11:03
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066069/

[7066072] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:18:04
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066072/

[7066073] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:30:48
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066073/

[7066074] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:32:30
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066074/

[7066076] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi}
time:   00:31:04
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066076/

[7066077] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:13:19
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066077/

[7066078] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/basic 3-final}
time:   00:09:45
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066078/

[7066082] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:18:33
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066082/

[7066083] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:10:35
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066083/

[7066084] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm_repos}
time:   00:03:42
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066084/

[7066085] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:20:19
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066085/

[7066086] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:13:16
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066086/

[7066087] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:11:31
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066087/

[7066090] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_iscsi_pids_limit/{centos_8.stream_container_tools test_iscsi_pids_limit}}
time:   00:12:23
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066090/

[7066091] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:11:04
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066091/

[7066092] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:16:29
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066092/

[7066094] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:19:38
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066094/

[7066095] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:18:21
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066095/

[7066096] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_nfs}
time:   00:25:42
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066096/

[7066097] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:34:54
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066097/

[7066098] orch/cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root}
time:   00:41:54
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066098/

[7066099] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys}
time:   00:35:18
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066099/

[7066100] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:17:19
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066100/

[7066101] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:29:24
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066101/

[7066102] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:15:22
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066102/

[7066103] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/basic 3-final}
time:   00:16:01
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066103/

[7066105] orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:17:33
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066105/

[7066107] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:14:40
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066107/

[7066108] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/on orchestrator_cli}
time:   00:13:01
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066108/

[7066109] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/rgw 3-final}
time:   00:10:42
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066109/

[7066110] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root}
time:   00:34:04
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066110/

[7066112] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests}
time:   00:33:26
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066112/

[7066113] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/iscsi 3-final}
time:   00:11:26
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066113/

[7066114] orch/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:15:26
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066114/

[7066116] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/jaeger 3-final}
time:   00:16:49
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066116/

[7066117] orch/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:21:44
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066117/

[7066118] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:16:24
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066118/

[7066120] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root}
time:   00:34:44
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066120/

[7066121] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python}
time:   00:22:09
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066121/

[7066122] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_adoption}
time:   00:07:05
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066122/

[7066123] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
time:   00:18:46
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066123/

[7066124] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:14:58
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066124/

[7066125] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:29:35
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066125/

[7066126] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:13:05
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066126/

[7066128] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:12:46
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066128/

[7066130] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:13:46
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066130/

[7066133] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:11:19
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066133/

[7066134] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:19:02
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066134/

[7066135] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:13:03
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066135/

[7066136] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_cephadm_repos}
time:   00:04:47
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066136/

[7066138] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:18:18
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066138/

[7066139] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:34:12
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066139/

[7066140] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/off orchestrator_cli}
time:   00:12:37
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066140/

[7066141] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi}
time:   00:28:30
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066141/

[7066142] orch/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 1-start 2-services/basic 3-final}
time:   00:14:31
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066142/

[7066146] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:16:31
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066146/

[7066147] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:11:30
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066147/

[7066150] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:15:16
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066150/

[7066151] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:29:06
info:   http://pulpito.ceph.com/yuriw-2022-10-13_17:36:58-orch-main-distro-default-smithi/7066151/