2023-04-27T13:36:15.416 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-04-27T13:36:15.417 INFO:teuthology.results:subset: '1/8'
2023-04-27T13:36:15.418 INFO:teuthology.results:seed: '5257'
2023-04-27T19:10:54.465 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-04-27T19:10:54.491 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-28T06:50:53.526 INFO:teuthology.results:Tests finished! gathering results...
2023-04-28T06:51:01.557 INFO:scrape:Found 104 jobs
2023-04-28T06:51:14.488 INFO:scrape:Found 9 distinct failure reasons
2023-04-28T06:51:14.489 INFO:scrape:Failure: Command failed on smithi006 with status 1: 'sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e30f1e5e-e516-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4'
2023-04-28T06:51:14.489 INFO:scrape:55 jobs
2023-04-28T06:51:14.490 INFO:scrape:suites intersection: []
2023-04-28T06:51:14.491 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/basic', '2-services/rgw', '2-thrash', '3-final}', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'agent/off', 'agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke/{0-distro/ubuntu_20.04', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'root}', 'start', 'start}', 'task/test_iscsi_container/{centos_8.stream_container_tools', 'task/test_nfs}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'test_iscsi_container}}']
2023-04-28T06:51:14.491 INFO:scrape:
2023-04-28T06:51:14.492 INFO:scrape:Failure: timeout expired in wait_for_all_osds_up
2023-04-28T06:51:14.492 INFO:scrape:22 jobs: ['7251077', '7251084', '7251098', '7251110', '7251038', '7251029', '7251072', '7251064', '7251075', '7251096', '7251044', '7251033', '7251108', '7251101', '7251053', '7251059', '7251027', '7251021', '7251011', '7251088', '7251057', '7251046']
2023-04-28T06:51:14.493 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start']
2023-04-28T06:51:14.494 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/jaeger', '2-services/mirror', '2-services/nfs', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs2', '2-services/rgw', '2-services/rgw-ingress', '3-final}', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/osds/{0-distro/ubuntu_20.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04']
2023-04-28T06:51:14.495 INFO:scrape:
2023-04-28T06:51:14.495 INFO:scrape:Failure: {'smithi033.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}}
2023-04-28T06:51:14.496 INFO:scrape:3 jobs: ['7251028', '7251085', '7251015']
2023-04-28T06:51:14.496 INFO:scrape:suites intersection: []
2023-04-28T06:51:14.497 INFO:scrape:suites union: ['1-bootstrap/16.2.5', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'base/install', 'cluster/{fixed-3', 'fixed-2', 'mode/root', 'mon_election/connectivity', 'msgr/async-v1only', 'openstack}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'start', 'tasks/rados_python}', 'workloads/cephadm_iscsi}']
2023-04-28T06:51:14.498 INFO:scrape:
2023-04-28T06:51:14.498 INFO:scrape:Dead: b'2023-04-28T02:30:37.077 DEBUG:teuthology.exit:Finished running handlers'
2023-04-28T06:51:14.498 INFO:scrape:5 jobs: ['7251040', '7251048', '7251012', '7251092', '7251013']
2023-04-28T06:51:14.499 INFO:scrape:suites intersection: []
2023-04-28T06:51:14.499 INFO:scrape:suites union: ['1-bootstrap/16.2.0', '1-bootstrap/16.2.4', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '3-inline/no', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4']
2023-04-28T06:51:14.499 INFO:scrape:
2023-04-28T06:51:14.500 INFO:scrape:Failure: Command failed on smithi096 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 50b87cd8-e52c-11ed-9b00-001a4aab830c -e sha1=0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\''
2023-04-28T06:51:14.500 INFO:scrape:2 jobs: ['7251069', '7251018']
2023-04-28T06:51:14.500 INFO:scrape:suites intersection: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'mon_election/classic}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools']
2023-04-28T06:51:14.500 INFO:scrape:suites union: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'mon_election/classic}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools']
2023-04-28T06:51:14.501 INFO:scrape:
2023-04-28T06:51:14.501 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi110.front.sepia.ceph.com: ['type=AVC msg=audit(1682620650.192:20090): avc:  denied  { ioctl } for  pid=109887 comm="iptables" path="/var/lib/containers/storage/overlay/36e176a9b764059bb21bc52c53ddebc4c87b2bb9c14f5aac019caa7afa8b1cb3/merged" dev="overlay" ino=3279057 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1', 'type=AVC msg=audit(1682620650.348:20094): avc:  denied  { ioctl } for  pid=109914 comm="iptables" path="/var/lib/containers/storage/overlay/36e176a9b764059bb21bc52c53ddebc4c87b2bb9c14f5aac019caa7afa8b1cb3/merged" dev="overlay" ino=3279057 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1']
2023-04-28T06:51:14.501 INFO:scrape:1 jobs: ['7251083']
2023-04-28T06:51:14.502 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_cephadm}']
2023-04-28T06:51:14.502 INFO:scrape:
2023-04-28T06:51:14.502 INFO:scrape:Failure: Command failed on smithi155 with status 127: 'sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fea46e6a-e509-11ed-9b00-001a4aab830c -- ceph mon dump -f json'
2023-04-28T06:51:14.503 INFO:scrape:1 jobs: ['7251039']
2023-04-28T06:51:14.503 INFO:scrape:suites: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async-v1only', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'start', 'tasks/rados_python}']
2023-04-28T06:51:14.503 INFO:scrape:
2023-04-28T06:51:14.503 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi188.front.sepia.ceph.com: ['type=AVC msg=audit(1682613278.526:19263): avc:  denied  { ioctl } for  pid=152446 comm="iptables" path="/var/lib/containers/storage/overlay/256a4cc4e3ccd227ae4bb185509f6e16815ef9e002a6e08621102747e387d673/merged" dev="overlay" ino=3411199 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1']
2023-04-28T06:51:14.504 INFO:scrape:1 jobs: ['7251043']
2023-04-28T06:51:14.504 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}']
2023-04-28T06:51:14.505 INFO:scrape:
2023-04-28T06:51:14.505 INFO:scrape:Failure: Command failed on smithi089 with status 1: 'sudo yum install -y kernel'
2023-04-28T06:51:14.506 INFO:scrape:1 jobs: ['7251009']
2023-04-28T06:51:14.507 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools']
2023-04-28T06:51:14.507 INFO:scrape:
2023-04-28T06:51:14.513 INFO:teuthology.results:Sending results to gabrioux@redhat.com: Test Run: gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/
failed:  83
dead:    8
running: 0
waiting: 0
queued:  0
passed:  13


Fail
=================================================================
[7251009]  orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251009/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251009/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3bf13c7a006b44f8b77ba540e2191d94

    Command failed on smithi089 with status 1: 'sudo yum install -y kernel'

[7251010]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:24:22
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251010/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251010/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a62a6e97683647aaa3ec9fd1a3c64670

    Command failed on smithi033 with status 1: 'sudo cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    c6db1dda-e504-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251011]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/jaeger 3-final}
-----------------------------------------------------------------
time:   00:47:27
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251011/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251011/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d497829ae92842d5a0640b9f2d67da5a

    timeout expired in wait_for_all_osds_up

[7251016]  orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/basic 3-final}
-----------------------------------------------------------------
time:   00:11:32
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251016/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251016/
sentry: https://sentry.ceph.com/organizations/ceph/?query=61acc9a303bd41f8b20bb61695976ca0

    Command failed on smithi150 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    d6363aea-e503-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251017]  orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:39:57
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251017/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251017/
sentry: https://sentry.ceph.com/organizations/ceph/?query=45e96a1979d243f4ac2e20205a290083

    Command failed on smithi003 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    5538905c-e50b-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1'

[7251018]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   01:24:09
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251018/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251018/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9948a54e46204bc7b0596230b909b7c8

    Command failed on smithi086 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    e451a47c-e50b-11ed-9b00-001a4aab830c -e
    sha1=0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.osd | length == 2\'"\'"\'\''

[7251019]  orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_nfs}
-----------------------------------------------------------------
time:   01:25:00
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251019/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251019/
sentry: https://sentry.ceph.com/organizations/ceph/?query=29ba8874cf9440f1b0efdddebc139f8c

    Command failed on smithi141 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    606e56c6-e512-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251020]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:20:12
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251020/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251020/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3905c9425222421c9e6631abd245d729

    Command failed on smithi044 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    1794c04a-e506-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251021]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/mirror 3-final}
-----------------------------------------------------------------
time:   00:21:03
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251021/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251021/
sentry: https://sentry.ceph.com/organizations/ceph/?query=050e4d45164b41ea9f0783dd9fff3832

    timeout expired in wait_for_all_osds_up

[7251022]  orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_orch_cli}
-----------------------------------------------------------------
time:   01:29:31
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251022/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251022/
sentry: https://sentry.ceph.com/organizations/ceph/?query=44983c84bea4467e8e3805a4698ec96e

    Command failed on smithi023 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    cc2650fc-e513-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251023]  orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:14:19
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251023/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251023/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a557803c6b5a4d50baf55c23e929e78f

    Command failed on smithi077 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    0f531d64-e506-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1'

[7251024]  orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:15:32
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251024/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251024/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cb96c0ba6fb449a9aa2c82d26239818b

    Command failed on smithi031 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    ca67bce0-e506-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251025]  orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:13:45
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251025/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251025/
sentry: https://sentry.ceph.com/organizations/ceph/?query=567129378b7141f89cd2254ef725b498

    Command failed on smithi159 with status 1: 'sudo cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    7daefe8a-e507-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251026]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_orch_cli_mon}
-----------------------------------------------------------------
time:   00:26:12
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251026/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251026/
sentry: https://sentry.ceph.com/organizations/ceph/?query=78c629bb649e4d74bf25f594e7e2335f

    Command failed on smithi039 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    0abac600-e509-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251027]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
-----------------------------------------------------------------
time:   00:19:07
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251027/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251027/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ffcc698ca7054004922bfa680f3c1811

    timeout expired in wait_for_all_osds_up

[7251029]  orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag}
-----------------------------------------------------------------
time:   00:47:57
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251029/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251029/
sentry: https://sentry.ceph.com/organizations/ceph/?query=06001b5e955b41d0863d7b97a5fa2957

    timeout expired in wait_for_all_osds_up

[7251031]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_rgw_multisite}
-----------------------------------------------------------------
time:   00:24:09
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251031/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251031/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0d690b0bf6a040a0b84bc80018ef6358

    Command failed on smithi097 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    a427d404-e509-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251032]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   01:32:27
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251032/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251032/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b750ff8728cb4b7eaca6168df442618a

    Command failed on smithi006 with status 1: 'sudo cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    e30f1e5e-e516-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251033]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
-----------------------------------------------------------------
time:   00:20:40
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251033/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251033/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca00660c7bca40be914b9bafc2edf1f7

    timeout expired in wait_for_all_osds_up

[7251034]  orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:12:24
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251034/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251034/
sentry: https://sentry.ceph.com/organizations/ceph/?query=754946d37ca44ed9893ad24f875db6e2

    Command failed on smithi077 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    5f2f9dbe-e509-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1'

[7251035]  orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:14:31
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251035/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251035/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9acd66399f614662a179d06d06dc4d09

    Command failed on smithi038 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    bc696b04-e509-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251036]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:20:37
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251036/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251036/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cdd686a806174568a17bc618bf8289e9

    Command failed on smithi044 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    e9907df2-e509-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251038]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
-----------------------------------------------------------------
time:   00:47:36
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251038/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251038/
sentry: https://sentry.ceph.com/organizations/ceph/?query=509a3dde52f9492da4352d63c1b99910

    timeout expired in wait_for_all_osds_up

[7251039]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:16:59
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251039/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251039/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ad0c0c20bb2046f9a66d66ac202f84f7

    Command failed on smithi155 with status 127: 'sudo cephadm --image quay-
    quay-quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    fea46e6a-e509-11ed-9b00-001a4aab830c -- ceph mon dump -f json'

[7251041]  orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:39:25
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251041/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251041/
sentry: https://sentry.ceph.com/organizations/ceph/?query=de12c1ca9d1742338ebd478b1334c67d

    Command failed on smithi089 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    d08656ea-e510-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1'

[7251043]  orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_cephadm}
-----------------------------------------------------------------
time:   01:36:56
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251043/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251043/

    SELinux denials found on ubuntu@smithi188.front.sepia.ceph.com: ['type=AVC
    msg=audit(1682613278.526:19263): avc:  denied  { ioctl } for  pid=152446
    comm="iptables" path="/var/lib/containers/storage/overlay/256a4cc4e3ccd227a
    e4bb185509f6e16815ef9e002a6e08621102747e387d673/merged" dev="overlay"
    ino=3411199 scontext=system_u:system_r:iptables_t:s0
    tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir
    permissive=1']

[7251044]  orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait}
-----------------------------------------------------------------
time:   00:20:37
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251044/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251044/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6a6f4c4796c842a3aca10e87fa9e7b38

    timeout expired in wait_for_all_osds_up

[7251045]  orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:13:33
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251045/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251045/
sentry: https://sentry.ceph.com/organizations/ceph/?query=101093bf0cee4d909148aa664f35578b

    Command failed on smithi053 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    9338a748-e50b-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251046]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
-----------------------------------------------------------------
time:   00:47:25
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251046/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251046/
sentry: https://sentry.ceph.com/organizations/ceph/?query=de09861b43874b309ec99d5db01a92a7

    timeout expired in wait_for_all_osds_up

[7251049]  orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:39:17
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251049/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251049/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ce251a2df5ea471e881827cf53828a9e

    Command failed on smithi012 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    36199dc2-e512-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1'

[7251050]  orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   01:29:43
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251050/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251050/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c48b3c2f96fe48c8a2f2edc3bd9402a7

    Command failed on smithi088 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    916d41a4-e519-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251051]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   01:30:04
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251051/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251051/
sentry: https://sentry.ceph.com/organizations/ceph/?query=10b1cde9aece4299b3bd69d9173e9c56

    Command failed on smithi026 with status 1: 'sudo cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    d14ff03c-e519-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251052]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}}
-----------------------------------------------------------------
time:   01:31:10
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251052/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251052/
sentry: https://sentry.ceph.com/organizations/ceph/?query=91c19803619741fd885d6dc377444492

    Command failed on smithi187 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    06647dba-e52e-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251053]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final}
-----------------------------------------------------------------
time:   00:27:39
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251053/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251053/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9cecd4187fae4701848cafc60866c01c

    timeout expired in wait_for_all_osds_up

[7251055]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python}
-----------------------------------------------------------------
time:   01:32:14
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251055/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251055/
sentry: https://sentry.ceph.com/organizations/ceph/?query=47b7324b373141bfa1e04043c0b748cb

    Command failed on smithi073 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    fb0b9858-e52e-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251056]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_nfs}
-----------------------------------------------------------------
time:   00:14:25
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251056/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251056/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6c378fc1895a4f4a8425bef111e0c8ee

    Command failed on smithi163 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    4bc9a8c4-e521-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251057]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final}
-----------------------------------------------------------------
time:   00:19:51
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251057/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251057/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4a2874ffdaeb4f06986b8df15c267cf7

    timeout expired in wait_for_all_osds_up

[7251058]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:19:48
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251058/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251058/
sentry: https://sentry.ceph.com/organizations/ceph/?query=53e4ebbbbc71410183722244daec1413

    Command failed on smithi105 with status 1: 'sudo cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    5dfa9e08-e522-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251059]  orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
-----------------------------------------------------------------
time:   00:19:24
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251059/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251059/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3d007329d7e744818b5a06e49a752b1d

    timeout expired in wait_for_all_osds_up

[7251060]  orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_orch_cli}
-----------------------------------------------------------------
time:   00:11:29
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251060/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251060/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a68c145542de430abb9858354d36f0b9

    Command failed on smithi038 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    4a6d520e-e522-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251061]  orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:41:36
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251061/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251061/
sentry: https://sentry.ceph.com/organizations/ceph/?query=df0d761582de412f9b736b57840869cd

    Command failed on smithi044 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    d45a6e1e-e529-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1'

[7251062]  orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   01:30:24
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251062/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251062/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0bb338ca07e94e0d9543faeb611e3ec9

    Command failed on smithi084 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    55d13c46-e531-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251063]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:17:02
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251063/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251063/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a319249bfea349b29ed679f9b3cdc6ad

    Command failed on smithi088 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    60ab2814-e524-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251064]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
-----------------------------------------------------------------
time:   00:20:34
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251064/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251064/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2d76b6f69fb24705a12c2d1927583ac7

    timeout expired in wait_for_all_osds_up

[7251067]  orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/rgw 3-final}
-----------------------------------------------------------------
time:   00:38:54
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251067/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251067/
sentry: https://sentry.ceph.com/organizations/ceph/?query=db904baadf274662883e75c3bea2ffe8

    Command failed on smithi086 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    8b66fb94-e52b-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251068]  orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:39:35
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251068/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251068/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4f93e9c6d11b49bdbe47ab94fff5d593

    Command failed on smithi038 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    2b2df7c2-e52c-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1'

[7251069]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic}
-----------------------------------------------------------------
time:   01:24:13
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251069/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251069/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c382dc890f4147b4ac72246773426e74

    Command failed on smithi096 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    50b87cd8-e52c-11ed-9b00-001a4aab830c -e
    sha1=0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.osd | length == 2\'"\'"\'\''

[7251070]  orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli_mon}
-----------------------------------------------------------------
time:   01:42:59
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251070/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251070/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8c48f541a98d4618a90ca4b47558214e

    Command failed on smithi002 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    cf261932-e535-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251071]  orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python}
-----------------------------------------------------------------
time:   00:11:58
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251071/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251071/
sentry: https://sentry.ceph.com/organizations/ceph/?query=94e14f2ad42547ad9562cbe213bdaf71

    Command failed on smithi162 with status 1: 'sudo cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    4a0a3fbc-e526-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251072]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw 3-final}
-----------------------------------------------------------------
time:   00:47:59
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251072/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251072/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ddb7a7f29d434517b5370da4791dbd05

    timeout expired in wait_for_all_osds_up

[7251073]  orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_rgw_multisite}
-----------------------------------------------------------------
time:   01:35:10
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251073/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251073/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d269c57df4aa457bbb949dcab6ef5caa

    Command failed on smithi115 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    0ae094d0-e535-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251074]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   01:28:34
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251074/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251074/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e95a999b73b64ba7a8fc8243ff324a1a

    Command failed on smithi088 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    fba4d1e8-e534-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251075]  orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all}
-----------------------------------------------------------------
time:   00:19:42
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251075/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251075/
sentry: https://sentry.ceph.com/organizations/ceph/?query=14bc0c73846f4453935231c7f81a5493

    timeout expired in wait_for_all_osds_up

[7251076]  orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:11:50
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251076/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251076/
sentry: https://sentry.ceph.com/organizations/ceph/?query=678bcbd35e8049c5bd0e55f4ba84737a

    Command failed on smithi039 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    718ff4e0-e527-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1'

[7251077]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/basic 3-final}
-----------------------------------------------------------------
time:   00:51:49
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251077/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251077/
sentry: https://sentry.ceph.com/organizations/ceph/?query=32cd440b6f9c4e9b87e513667b228189

    timeout expired in wait_for_all_osds_up

[7251078]  orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:24:14
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251078/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251078/
sentry: https://sentry.ceph.com/organizations/ceph/?query=da628573dcd54a9780fcd57671faf825

    Command failed on smithi090 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    75d18dea-e52a-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251079]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   01:36:21
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251079/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251079/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cf0dbf2bd85a442aabc45bb45bd3de54

    Command failed on smithi033 with status 1: 'sudo cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    32be732a-e538-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251082]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python}
-----------------------------------------------------------------
time:   00:22:12
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251082/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251082/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d51a0da17a9142afae9943968ed80880

    Command failed on smithi114 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    b21251ea-e52a-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251083]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_cephadm}
-----------------------------------------------------------------
time:   00:20:00
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251083/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251083/

    SELinux denials found on ubuntu@smithi110.front.sepia.ceph.com: ['type=AVC
    msg=audit(1682620650.192:20090): avc:  denied  { ioctl } for  pid=109887
    comm="iptables" path="/var/lib/containers/storage/overlay/36e176a9b764059bb
    21bc52c53ddebc4c87b2bb9c14f5aac019caa7afa8b1cb3/merged" dev="overlay"
    ino=3279057 scontext=system_u:system_r:iptables_t:s0
    tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir
    permissive=1', 'type=AVC msg=audit(1682620650.348:20094): avc:  denied  {
    ioctl } for  pid=109914 comm="iptables" path="/var/lib/containers/storage/o
    verlay/36e176a9b764059bb21bc52c53ddebc4c87b2bb9c14f5aac019caa7afa8b1cb3/mer
    ged" dev="overlay" ino=3279057 scontext=system_u:system_r:iptables_t:s0
    tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir
    permissive=1']

[7251084]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/client-keyring 3-final}
-----------------------------------------------------------------
time:   00:20:48
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251084/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251084/
sentry: https://sentry.ceph.com/organizations/ceph/?query=81a251dde70b4b3d99c702d7d2cbab04

    timeout expired in wait_for_all_osds_up

[7251086]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:21:22
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251086/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251086/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2c2cef6d75ad4944be1b88ee56c06c1d

    Command failed on smithi149 with status 1: 'sudo cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    70ae53a0-e52c-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251088]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/iscsi 3-final}
-----------------------------------------------------------------
time:   00:19:41
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251088/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251088/
sentry: https://sentry.ceph.com/organizations/ceph/?query=edb592496ff040a1968367020dbe71f8

    timeout expired in wait_for_all_osds_up

[7251089]  orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:11:09
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251089/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251089/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f9e6d9160145495a85e35a45ed6d5e25

    Command failed on smithi081 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    ef8578c6-e52b-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1'

[7251090]  orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:14:51
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251090/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251090/
sentry: https://sentry.ceph.com/organizations/ceph/?query=649762e3c6524b9e9379122c4e0d3b77

    Command failed on smithi129 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    81654172-e52c-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251091]  orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:11:40
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251091/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251091/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a02cf309169645429f3ae112a8392df5

    Command failed on smithi044 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    c78ec5c4-e52c-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251093]  orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:39:36
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251093/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251093/
sentry: https://sentry.ceph.com/organizations/ceph/?query=024a76f28bec402c9387911b0f2b9c38

    Command failed on smithi006 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    e81b4ce2-e534-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1'

[7251095]  orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}}
-----------------------------------------------------------------
time:   01:30:07
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251095/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251095/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cdb8e52164eb43c793c7a88945d2d2fe

    Command failed on smithi090 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    264f8292-e53c-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251096]  orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:20:02
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251096/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251096/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7ce1083a072b4467b66e6173e70a996d

    timeout expired in wait_for_all_osds_up

[7251097]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python}
-----------------------------------------------------------------
time:   01:30:17
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251097/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251097/
sentry: https://sentry.ceph.com/organizations/ceph/?query=18b2d9b117034f91be0725f170820bfc

    Command failed on smithi086 with status 1: 'sudo cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    37ddd86a-e53c-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251098]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/jaeger 3-final}
-----------------------------------------------------------------
time:   00:20:20
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251098/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251098/
sentry: https://sentry.ceph.com/organizations/ceph/?query=446d5fa421d4459d8154234e609432e0

    timeout expired in wait_for_all_osds_up

[7251099]  orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_nfs}
-----------------------------------------------------------------
time:   01:27:39
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251099/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251099/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cc793bafbd27412a87270270435cf88c

    Command failed on smithi196 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    3a07a08a-e53c-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251100]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   01:32:24
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251100/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251100/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dd9daf9059c048d6bfa1c69b3abf4466

    Command failed on smithi081 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    3e532bcc-e53d-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251101]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/mirror 3-final}
-----------------------------------------------------------------
time:   00:48:31
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251101/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251101/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bf00626a093e43f6a1d33cbaf9f13486

    timeout expired in wait_for_all_osds_up

[7251102]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_orch_cli}
-----------------------------------------------------------------
time:   00:21:36
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251102/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251102/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8425d8568b174829b69f7587006d3b91

    Command failed on smithi136 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    3d8f0d4e-e530-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251103]  orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:09:44
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251103/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251103/
sentry: https://sentry.ceph.com/organizations/ceph/?query=01f6f95f65c24f0aa5285c7dccdd764f

    Command failed on smithi149 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    f76498d0-e52e-11ed-9b00-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1'

[7251104]  orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:11:42
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251104/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251104/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d461d45d0e244b5cb18459c3e1db7d1f

    Command failed on smithi172 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    5caa8ed4-e52f-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251105]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:16:15
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251105/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251105/
sentry: https://sentry.ceph.com/organizations/ceph/?query=52686a70c7b1468ba5b24d2c2b927d12

    Command failed on smithi044 with status 1: 'sudo cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    2782641a-e530-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251107]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_orch_cli_mon}
-----------------------------------------------------------------
time:   00:23:15
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251107/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251107/
sentry: https://sentry.ceph.com/organizations/ceph/?query=acad83d47f4b47fcb5d57d0f4b9af32a

    Command failed on smithi087 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    c50fd446-e531-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251108]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
-----------------------------------------------------------------
time:   00:47:16
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251108/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251108/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0cc71a3629e44e9a94cb2a63ed42b7f2

    timeout expired in wait_for_all_osds_up

[7251109]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python}
-----------------------------------------------------------------
time:   00:16:44
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251109/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251109/
sentry: https://sentry.ceph.com/organizations/ceph/?query=93f740d3c9784790a2446c6ccea822c9

    Command failed on smithi031 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    4fb8f880-e531-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251110]  orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag}
-----------------------------------------------------------------
time:   00:47:35
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251110/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251110/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e866198ad17498395c807c87fabe2bf

    timeout expired in wait_for_all_osds_up

[7251111]  orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_rgw_multisite}
-----------------------------------------------------------------
time:   00:12:33
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251111/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251111/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8d8965b9ee734e9bb38cc59803ced79e

    Command failed on smithi019 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    d0e00f66-e531-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'

[7251112]  orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:11:48
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251112/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251112/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9b8ce4d40170428898e30e0cd001f89d

    Command failed on smithi120 with status 1: 'sudo cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:0fac42ee00164c4a9a3aab987f72c8d170d5bcb7 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    d1c3ea7e-e531-11ed-9b00-001a4aab830c -- ceph-volume lvm zap
    /dev/vg_nvme/lv_4'



Dead
=================================================================
[7251012]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251012/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251012/

    hit max job timeout

[7251013]  orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251013/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251013/

    hit max job timeout

[7251015]  orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi}
-----------------------------------------------------------------
time:   00:34:11
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251015/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251015/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1a0aecdab8e742f2bf5f09e6256bc100

    {'smithi148.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/144bb6d03f4cceaf
    da5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot
    download, all mirrors were already tried without success", 'rc': 1,
    'results': []}}

[7251028]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:34:06
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251028/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251028/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e6e5b68776c45ecbb183d4506b5fc77

    {'smithi033.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s):
    repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-
    comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already
    tried without success", 'rc': 1, 'results': []}}

[7251040]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251040/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251040/

    hit max job timeout

[7251048]  orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251048/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251048/

    hit max job timeout

[7251085]  orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:34:08
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251085/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251085/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8f51e66b8d1c4c91b6d14aa6667d7a4d

    {'smithi183.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/144bb6d03f4cceaf
    da5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot
    download, all mirrors were already tried without success", 'rc': 1,
    'results': []}}

[7251092]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251092/
log:    http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251092/

    hit max job timeout



Pass
=================================================================
[7251014] orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0} 2-node-mgr agent/off orchestrator_cli}
time:   00:20:19
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251014/

[7251030] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:45:12
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251030/

[7251037] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_adoption}
time:   00:10:42
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251037/

[7251042] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
time:   00:36:14
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251042/

[7251047] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_cephadm_repos}
time:   00:38:08
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251047/

[7251054] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:48:17
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251054/

[7251065] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:48:23
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251065/

[7251066] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/on orchestrator_cli}
time:   01:37:34
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251066/

[7251080] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_adoption}
time:   00:15:01
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251080/

[7251081] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:52:50
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251081/

[7251087] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_cephadm_repos}
time:   00:07:04
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251087/

[7251094] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
time:   00:34:13
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251094/

[7251106] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:49:20
info:   http://pulpito.ceph.com/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251106/