2023-04-04T02:09:15.122 INFO:root:teuthology version: 0.0.1.dev93+gd7c4062
2023-04-04T02:09:15.123 INFO:teuthology.results:subset: '1/13'
2023-04-04T02:09:15.123 INFO:teuthology.results:seed: '8979'
2023-04-04T03:09:27.982 INFO:root:teuthology version: 0.0.1.dev93+gd7c4062
2023-04-04T03:09:27.986 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-04T05:00:37.057 INFO:teuthology.results:Tests finished! gathering results...
2023-04-04T05:00:43.063 INFO:scrape:Found 98 jobs
2023-04-04T05:00:43.536 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230481/teuthology.log
2023-04-04T05:00:44.339 INFO:scrape:Found 5 distinct failure reasons
2023-04-04T05:00:44.340 INFO:scrape:Failure: Command failed on smithi196 with status 1: 'sudo yum install -y kernel'
2023-04-04T05:00:44.340 INFO:scrape:5 jobs: ['7230495', '7230500', '7230437', '7230482', '7230504']
2023-04-04T05:00:44.341 INFO:scrape:suites intersection: []
2023-04-04T05:00:44.341 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/16.2.5', '1-start', '2-nfs', '2-services/nfs-keepalive-only', '3-final}', '3-upgrade-with-workload', '4-final}', 'agent/on', 'fixed-2', 'mode/root', 'mon_election/connectivity', 'msgr/async', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'start', 'task/test_orch_cli_mon}', 'tasks/rados_api_tests}']
2023-04-04T05:00:44.341 INFO:scrape:
2023-04-04T05:00:44.342 INFO:scrape:Failure: Command failed on smithi012 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:27e1ef4e404073fe46b21aac3f1ae821adacafcc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6d3e13ac-d291-11ed-9aff-001a4aab830c -- bash -c \'set -ex\n# since we don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in order to replace the fake hostnames "host.a/b/c" with\n# the actual names cephadm knows the host by within the mon spec\nceph orch host ls --format json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'{datacenter=a}\\n{datacenter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' > crush_locs\nceph orch ls --service-name mon --export > mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3 && read fakename <&4; do\n  MONSPEC="${MONSPEC//$fakename/$realname}"\ndone 3<realnames 4<fakenames\necho "$MONSPEC" > mon.yaml\ncat mon.yaml\n# now the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep 90\nceph mon dump\nceph mon dump --format json\n# verify all the crush locations got set from "ceph mon dump" output\nwhile read hostname <&3 && read crushloc <&4; do\n  ceph mon dump --format json | jq --arg hostname "$hostname" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] | select(.name == $hostname) | .crush_location == $crushloc\'"\'"\'\ndone 3<realnames 4<crush_locs\n\''
2023-04-04T05:00:44.342 INFO:scrape:2 jobs: ['7230453', '7230516']
2023-04-04T05:00:44.342 INFO:scrape:suites intersection: ['agent/on', 'mon_election/connectivity', 'task/test_set_mon_crush_locations}']
2023-04-04T05:00:44.343 INFO:scrape:suites union: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_set_mon_crush_locations}']
2023-04-04T05:00:44.343 INFO:scrape:
2023-04-04T05:00:44.343 INFO:scrape:Failure: Command failed on smithi067 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58c56396-d2a3-11ed-9aff-001a4aab830c -- ceph orch daemon add osd smithi067:vg_nvme/lv_4'
2023-04-04T05:00:44.344 INFO:scrape:3 jobs: ['7230517', '7230451', '7230512']
2023-04-04T05:00:44.344 INFO:scrape:suites intersection: []
2023-04-04T05:00:44.344 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-repo_digest/defaut', '3-inline/no', '3-upgrade-mgr-staggered', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4']
2023-04-04T05:00:44.344 INFO:scrape:
2023-04-04T05:00:44.345 INFO:scrape:Dead: None
2023-04-04T05:00:44.345 INFO:scrape:6 jobs: ['7230481', '7230443', '7230471', '7230424', '7230448', '7230462']
2023-04-04T05:00:44.345 INFO:scrape:
2023-04-04T05:00:44.346 INFO:scrape:Failure: Command failed on smithi046 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:27e1ef4e404073fe46b21aac3f1ae821adacafcc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 64eaa2b4-d297-11ed-9aff-001a4aab830c -- ceph osd stat -f json'
2023-04-04T05:00:44.346 INFO:scrape:2 jobs: ['7230484', '7230435']
2023-04-04T05:00:44.346 INFO:scrape:suites intersection: ['0-nvme-loop']
2023-04-04T05:00:44.347 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-flag}', 'agent/off', 'fixed-2', 'mon_election/classic', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'start}']
2023-04-04T05:00:44.347 INFO:scrape:
2023-04-04T05:00:44.351 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/
failed:  12
dead:    6
running: 0
waiting: 0
queued:  0
passed:  80


Fail
=================================================================
[7230435]  orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:12:13
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230435/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230435/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4faaf072495d473da22dba258b6b349d

    Command failed on smithi158 with status 127: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:27e1ef4e404073fe46b21aac3f1ae821adacafcc shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    52de402c-d291-11ed-9aff-001a4aab830c -- ceph mon dump -f json'

[7230437]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
-----------------------------------------------------------------
time:   00:19:58
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230437/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230437/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cee769b24d7c41faa4ac9b92e511a553

    Command failed on smithi038 with status 32: "sudo
    TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.38:/fake
    /mnt/foo'"

[7230451]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic}
-----------------------------------------------------------------
time:   00:37:59
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230451/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230451/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c2ad7b9cdda64c6a8b8b96e3b3e4f2c3

    Command failed on smithi100 with status 2: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    acfb2df4-d296-11ed-9aff-001a4aab830c -e
    sha1=27e1ef4e404073fe46b21aac3f1ae821adacafcc -- bash -c \'ceph orch ps
    --format json | jq --arg TARGET_ID "$TARGET_ID" -e \'"\'"\'.[] |
    select(.daemon_type=="mgr") |
    select(.container_image_id==$TARGET_ID)\'"\'"\' | jq -s > out.json\''

[7230453]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_set_mon_crush_locations}
-----------------------------------------------------------------
time:   00:23:01
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230453/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230453/
sentry: https://sentry.ceph.com/organizations/ceph/?query=50ac5921a7444ce5865dfd5f3bd239dc

    Command failed on smithi012 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:27e1ef4e404073fe46b21aac3f1ae821adacafcc shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    6d3e13ac-d291-11ed-9aff-001a4aab830c -- bash -c \'set -ex\n# since we
    don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in
    order to replace the fake hostnames "host.a/b/c" with\n# the actual names
    cephadm knows the host by within the mon spec\nceph orch host ls --format
    json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho
    $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'{datac
    enter=a}\\n{datacenter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' >
    crush_locs\nceph orch ls --service-name mon --export >
    mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3
    && read fakename <&4; do\n  MONSPEC="${MONSPEC//$fakename/$realname}"\ndone
    3<realnames 4<fakenames\necho "$MONSPEC" > mon.yaml\ncat mon.yaml\n# now
    the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph
    orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls
    --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep
    90\nceph mon dump\nceph mon dump --format json\n# verify all the crush
    locations got set from "ceph mon dump" output\nwhile read hostname <&3 &&
    read crushloc <&4; do\n  ceph mon dump --format json | jq --arg hostname
    "$hostname" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] |
    select(.name == $hostname) | .crush_location == $crushloc\'"\'"\'\ndone
    3<realnames 4<crush_locs\n\''

[7230482]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230482/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230482/
sentry: https://sentry.ceph.com/organizations/ceph/?query=13c0631c80144bb5b1675090bf71e012

    Command failed on smithi090 with status 1: 'sudo yum install -y kernel'

[7230484]  orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag}
-----------------------------------------------------------------
time:   00:28:13
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230484/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230484/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f8d4e54df60c4d3aa6870a40d414092d

    Command failed on smithi046 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:27e1ef4e404073fe46b21aac3f1ae821adacafcc shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    64eaa2b4-d297-11ed-9aff-001a4aab830c -- ceph osd stat -f json'

[7230495]  orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230495/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230495/
sentry: https://sentry.ceph.com/organizations/ceph/?query=89e93cca511940f5a226f39050e2ba4b

    Command failed on smithi196 with status 1: 'sudo yum install -y kernel'

[7230500]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
-----------------------------------------------------------------
time:   00:20:36
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230500/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230500/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8f11ac05151b49ffa4c76e087123d86a

    Command failed on smithi012 with status 32: "sudo
    TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.12:/fake
    /mnt/foo'"

[7230504]  orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli_mon}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230504/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230504/
sentry: https://sentry.ceph.com/organizations/ceph/?query=286ffd09b1ca4d8d906e053d1080f8c6

    Command failed on smithi148 with status 1: 'sudo yum install -y kernel'

[7230512]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
-----------------------------------------------------------------
time:   00:23:40
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230512/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230512/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d392b232419d45ceb86dbba0e91ae471

    Command failed on smithi084 with status 2: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    6d7acc84-d296-11ed-9aff-001a4aab830c -e
    sha1=27e1ef4e404073fe46b21aac3f1ae821adacafcc -- bash -c \'ceph orch ps
    --format json | jq --arg TARGET_ID "$TARGET_ID" -e \'"\'"\'.[] |
    select(.daemon_type=="mgr") |
    select(.container_image_id==$TARGET_ID)\'"\'"\' | jq -s > out.json\''

[7230516]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_set_mon_crush_locations}
-----------------------------------------------------------------
time:   00:21:31
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230516/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230516/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca3f21ead1594ffc8bcc53ec2c130e8f

    Command failed on smithi047 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:27e1ef4e404073fe46b21aac3f1ae821adacafcc shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    d4da9160-d297-11ed-9aff-001a4aab830c -- bash -c \'set -ex\n# since we
    don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in
    order to replace the fake hostnames "host.a/b/c" with\n# the actual names
    cephadm knows the host by within the mon spec\nceph orch host ls --format
    json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho
    $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'{datac
    enter=a}\\n{datacenter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' >
    crush_locs\nceph orch ls --service-name mon --export >
    mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3
    && read fakename <&4; do\n  MONSPEC="${MONSPEC//$fakename/$realname}"\ndone
    3<realnames 4<fakenames\necho "$MONSPEC" > mon.yaml\ncat mon.yaml\n# now
    the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph
    orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls
    --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep
    90\nceph mon dump\nceph mon dump --format json\n# verify all the crush
    locations got set from "ceph mon dump" output\nwhile read hostname <&3 &&
    read crushloc <&4; do\n  ceph mon dump --format json | jq --arg hostname
    "$hostname" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] |
    select(.name == $hostname) | .crush_location == $crushloc\'"\'"\'\ndone
    3<realnames 4<crush_locs\n\''

[7230517]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:14:34
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230517/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230517/
sentry: https://sentry.ceph.com/organizations/ceph/?query=22c54c3edb6f43bfa8a235150200e67d

    Command failed on smithi067 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    58c56396-d2a3-11ed-9aff-001a4aab830c -- ceph orch daemon add osd
    smithi067:vg_nvme/lv_4'



Dead
=================================================================
[7230424]  orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230424/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230424/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds

[7230443]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230443/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230443/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds

[7230448]  orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230448/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230448/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds

[7230462]  orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230462/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230462/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds

[7230471]  orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230471/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230471/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds

[7230481]  orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230481/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230481/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds



Pass
=================================================================
[7230423] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:32:35
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230423/

[7230425] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_nfs}
time:   00:35:32
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230425/

[7230426] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:58:48
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230426/

[7230427] orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0} 2-node-mgr agent/off orchestrator_cli}
time:   00:20:08
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230427/

[7230428] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi}
time:   01:49:33
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230428/

[7230429] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/basic 3-final}
time:   00:28:14
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230429/

[7230430] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:29:23
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230430/

[7230431] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys}
time:   00:37:50
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230431/

[7230432] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
time:   00:34:16
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230432/

[7230433] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:19:49
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230433/

[7230434] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:17:24
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230434/

[7230436] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli}
time:   01:15:16
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230436/

[7230438] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:35:06
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230438/

[7230439] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root}
time:   01:35:21
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230439/

[7230440] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests}
time:   01:38:11
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230440/

[7230441] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_orch_cli_mon}
time:   01:33:29
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230441/

[7230442] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:20:10
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230442/

[7230444] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:30:12
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230444/

[7230445] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:18:07
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230445/

[7230446] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:24:34
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230446/

[7230447] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_rgw_multisite}
time:   00:22:35
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230447/

[7230449] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:32:42
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230449/

[7230450] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:29:40
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230450/

[7230452] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys}
time:   00:31:59
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230452/

[7230454] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:29:07
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230454/

[7230455] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:18:48
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230455/

[7230456] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:17:43
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230456/

[7230457] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:32:08
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230457/

[7230458] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/basic 3-final}
time:   00:17:49
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230458/

[7230459] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root}
time:   00:47:34
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230459/

[7230460] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
time:   00:41:31
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230460/

[7230461] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_adoption}
time:   00:16:41
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230461/

[7230463] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:19:30
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230463/

[7230464] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:35:49
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230464/

[7230465] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm}
time:   00:18:46
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230465/

[7230466] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
time:   00:31:43
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230466/

[7230467] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/iscsi 3-final}
time:   00:30:24
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230467/

[7230468] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root}
time:   00:53:35
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230468/

[7230469] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
time:   00:33:41
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230469/

[7230470] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:18:52
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230470/

[7230472] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_cephadm_repos}
time:   00:35:20
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230472/

[7230473] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/jaeger 3-final}
time:   00:30:45
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230473/

[7230474] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys}
time:   01:29:16
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230474/

[7230475] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:19:48
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230475/

[7230476] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:34:08
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230476/

[7230477] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}}
time:   01:19:54
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230477/

[7230478] orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_20.04} 2-node-mgr agent/on orchestrator_cli}
time:   00:13:10
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230478/

[7230479] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final}
time:   00:18:49
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230479/

[7230480] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:29:15
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230480/

[7230483] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
time:   00:18:38
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230483/

[7230485] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:19:55
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230485/

[7230486] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_monitoring_stack_basic}
time:   00:25:44
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230486/

[7230487] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:21:00
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230487/

[7230488] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python}
time:   00:30:06
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230488/

[7230489] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:32:06
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230489/

[7230490] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:32:37
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230490/

[7230491] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root}
time:   01:42:27
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230491/

[7230492] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}
time:   00:43:44
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230492/

[7230493] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_nfs}
time:   00:30:48
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230493/

[7230494] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys}
time:   00:27:36
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230494/

[7230496] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:32:05
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230496/

[7230497] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:29:46
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230497/

[7230498] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:16:17
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230498/

[7230499] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_orch_cli}
time:   00:18:48
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230499/

[7230501] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root}
time:   01:37:19
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230501/

[7230502] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
time:   00:43:59
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230502/

[7230503] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:29:12
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230503/

[7230505] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:34:53
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230505/

[7230506] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:16:23
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230506/

[7230507] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
time:   01:27:57
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230507/

[7230508] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:18:30
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230508/

[7230509] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:17:34
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230509/

[7230510] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:24:46
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230510/

[7230511] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_rgw_multisite}
time:   01:19:31
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230511/

[7230513] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root}
time:   00:50:10
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230513/

[7230514] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:33:33
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230514/

[7230515] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys}
time:   01:28:47
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230515/

[7230518] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:32:16
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230518/

[7230519] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:16:01
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230519/

[7230520] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:31:00
info:   http://pulpito.ceph.com/adking-2023-04-04_02:06:27-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230520/